-- Logs begin at Sun 2022-09-25 00:10:28 UTC, end at Sun 2022-09-25 01:17:38 UTC. -- Sep 25 00:40:26.202018 dv-cinder-master-3-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Sep 25 00:40:26.209287 dv-cinder-master-3-iscsi devstack@keystone.service[81372]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Sep 25 00:40:26.211304 dv-cinder-master-3-iscsi devstack@keystone.service[81372]: *** Starting uWSGI 2.0.18-debian (64bit) on [Sun Sep 25 00:40:26 2022] *** Sep 25 00:40:26.211304 dv-cinder-master-3-iscsi devstack@keystone.service[81372]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Sep 25 00:40:26.211304 dv-cinder-master-3-iscsi devstack@keystone.service[81372]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Sep 25 00:40:26.211304 dv-cinder-master-3-iscsi devstack@keystone.service[81372]: nodename: dv-cinder-master-3-iscsi Sep 25 00:40:26.211304 dv-cinder-master-3-iscsi devstack@keystone.service[81372]: machine: x86_64 Sep 25 00:40:26.211638 dv-cinder-master-3-iscsi devstack@keystone.service[81372]: clock source: unix Sep 25 00:40:26.211638 dv-cinder-master-3-iscsi devstack@keystone.service[81372]: pcre jit disabled Sep 25 00:40:26.211638 dv-cinder-master-3-iscsi devstack@keystone.service[81372]: detected number of CPU cores: 4 Sep 25 00:40:26.211638 dv-cinder-master-3-iscsi devstack@keystone.service[81372]: current working directory: / Sep 25 00:40:26.211638 dv-cinder-master-3-iscsi devstack@keystone.service[81372]: detected binary path: /usr/bin/uwsgi-core Sep 25 00:40:26.211638 dv-cinder-master-3-iscsi devstack@keystone.service[81372]: your processes number limit is 47680 Sep 25 00:40:26.211638 dv-cinder-master-3-iscsi devstack@keystone.service[81372]: your memory page size is 4096 bytes Sep 25 00:40:26.211638 dv-cinder-master-3-iscsi devstack@keystone.service[81372]: detected max file descriptor number: 2048 Sep 25 00:40:26.211638 dv-cinder-master-3-iscsi devstack@keystone.service[81372]: lock engine: pthread robust mutexes Sep 25 00:40:26.211638 dv-cinder-master-3-iscsi devstack@keystone.service[81372]: thunder lock: enabled Sep 25 00:40:26.212217 dv-cinder-master-3-iscsi devstack@keystone.service[81372]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Sep 25 00:40:26.212217 dv-cinder-master-3-iscsi devstack@keystone.service[81372]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Sep 25 00:40:26.253278 dv-cinder-master-3-iscsi devstack@keystone.service[81372]: Python main interpreter initialized at 0x562cf5a699c0 Sep 25 00:40:26.253278 dv-cinder-master-3-iscsi devstack@keystone.service[81372]: python threads support enabled Sep 25 00:40:26.253278 dv-cinder-master-3-iscsi devstack@keystone.service[81372]: your server socket listen backlog is limited to 100 connections Sep 25 00:40:26.253278 dv-cinder-master-3-iscsi devstack@keystone.service[81372]: your mercy for graceful operations on workers is 90 seconds Sep 25 00:40:26.253680 dv-cinder-master-3-iscsi devstack@keystone.service[81372]: mapped 403077 bytes (393 KB) for 2 cores Sep 25 00:40:26.253759 dv-cinder-master-3-iscsi devstack@keystone.service[81372]: *** Operational MODE: preforking *** Sep 25 00:40:26.254018 dv-cinder-master-3-iscsi devstack@keystone.service[81372]: *** uWSGI is running in multiple interpreter mode *** Sep 25 00:40:26.254018 dv-cinder-master-3-iscsi devstack@keystone.service[81372]: spawned uWSGI master process (pid: 81372) Sep 25 00:40:26.254225 dv-cinder-master-3-iscsi systemd[1]: Started Devstack devstack@keystone.service. Sep 25 00:40:26.254900 dv-cinder-master-3-iscsi devstack@keystone.service[81372]: spawned uWSGI worker 1 (pid: 81386, cores: 1) Sep 25 00:40:26.258567 dv-cinder-master-3-iscsi devstack@keystone.service[81372]: spawned uWSGI worker 2 (pid: 81387, cores: 1) Sep 25 00:40:26.258567 dv-cinder-master-3-iscsi devstack@keystone.service[81372]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Sep 25 00:40:30.599126 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 25 00:40:30.599126 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: from cryptography.utils import int_from_bytes Sep 25 00:40:30.599920 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 25 00:40:30.599920 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: from cryptography.utils import int_from_bytes Sep 25 00:40:30.600473 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 25 00:40:30.600473 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: from cryptography.utils import int_from_bytes Sep 25 00:40:30.600932 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 25 00:40:30.600932 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: from cryptography.utils import int_from_bytes Sep 25 00:40:30.650543 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Sep 25 00:40:30.650731 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Sep 25 00:40:30.650876 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Sep 25 00:40:30.650971 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] command line args: [] {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Sep 25 00:40:30.651073 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Sep 25 00:40:30.651160 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Sep 25 00:40:30.651265 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] command line args: [] {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Sep 25 00:40:30.651382 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ================================================================================ {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Sep 25 00:40:30.651596 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] admin_token = **** {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.651689 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Sep 25 00:40:30.651850 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] config_dir = [] {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.651992 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ================================================================================ {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Sep 25 00:40:30.652082 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.652276 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] config_source = [] {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.652363 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] admin_token = **** {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.652539 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] debug = True {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.652626 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] config_dir = [] {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.652707 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.652929 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.652929 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] config_source = [] {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.653046 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.653104 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] debug = True {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.653265 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.653358 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.653514 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.653602 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.653662 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.653730 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.653918 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.654004 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] list_limit = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.654118 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] log_config_append = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.654201 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.654308 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.654391 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] list_limit = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.654452 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] log_dir = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.654551 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] log_config_append = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.654622 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] log_file = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.654726 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.654857 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.654944 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] log_dir = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.655052 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] log_file = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.655120 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.655264 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.655343 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.655508 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.655595 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.655737 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.655816 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.655932 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.656010 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.656130 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.656213 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.656329 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.656407 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.656552 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.656737 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.656911 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.657080 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.657267 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.657437 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.657617 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.657792 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.657974 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.658118 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.658285 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.658453 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.658623 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.658734 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.658977 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.658977 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.659286 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.52/identity {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.659286 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] publish_errors = False {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.659595 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.52/identity {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.659736 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.659859 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] publish_errors = False {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.660007 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.660134 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.660248 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.660385 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.660499 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.660610 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.660742 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.660866 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.660986 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] standard_threads = False {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.661122 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.661244 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.661391 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] standard_threads = False {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.661506 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.661624 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.661737 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.661737 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] use_journal = False {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.661947 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.662125 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] use_json = False {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.662243 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.662355 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] use_stderr = False {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.662497 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] use_journal = False {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.662618 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] use_syslog = False {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.662738 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] use_json = False {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.662738 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] use_stderr = False {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.662969 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.663089 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] use_syslog = False {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.663208 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.663323 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 00:40:30.663434 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.663556 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.663670 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.663781 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.663913 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.664031 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.664149 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.664351 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.664476 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.664589 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.664701 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.664818 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.664937 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.665092 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.665236 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.665377 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.665492 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.665603 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.665721 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.665843 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.665957 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.666068 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.666185 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.666304 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.666417 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.666538 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.666658 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.666776 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.666889 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.667540 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.667696 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.667822 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.667937 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.668049 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.668168 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.668281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.668397 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.668587 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.668711 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.668825 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.668936 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.669054 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.669167 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.669311 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.669434 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.669548 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.669660 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.669773 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.669893 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.669893 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.670092 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.670203 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.670330 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.670446 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.670569 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.670691 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.670810 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.670922 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.671034 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.671145 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.671265 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.671378 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.671490 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.671601 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.671713 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.671824 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.671943 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.672057 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.672169 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.672280 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.672391 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.672502 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.672622 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.672737 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.672849 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.672960 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.673076 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.673195 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.673338 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.673460 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.673579 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.673698 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.673813 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.673929 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.674121 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.674243 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.674363 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.674477 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.674589 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.674701 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.674821 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.674942 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.675055 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.675167 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.675278 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.675390 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.675510 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.675623 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.675735 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.675846 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.675969 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.676083 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.676204 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.676301 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.676402 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.676497 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.676591 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.676685 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.676786 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.676882 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.676977 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.677112 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.677218 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.677394 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.677511 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.677614 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.677726 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.677839 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.677959 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.678082 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.678202 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.678322 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.678445 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.678549 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.678668 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.678790 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.678906 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.679035 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.679159 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.679274 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.679387 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.679506 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.679620 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.679620 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.679843 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.679946 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.680058 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.680058 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.680269 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.680269 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.680493 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.680493 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.680705 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.680818 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.680930 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.681051 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.681168 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.681295 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.681416 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.681416 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.681630 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.681630 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.681857 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.681857 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.682069 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.682183 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.682296 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.682416 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.682531 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.682679 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.683311 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.683425 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.683538 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.683830 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.683954 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.684070 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.684187 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.684302 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.684415 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.684541 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.684656 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.684770 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.684883 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.684997 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.685116 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.685239 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.685409 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.685409 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.685623 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.685747 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.685862 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.686050 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.686050 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.686274 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.686389 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.686503 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.686625 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.686748 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.686872 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.686987 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.687100 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.687213 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.687332 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.687448 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.687569 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.687684 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.687798 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.687917 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.688031 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.688145 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.688145 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.688357 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.688470 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.688583 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.688696 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.688808 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.688929 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.689044 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.689157 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.689283 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.689399 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.689514 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] auth.external = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.689660 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.689789 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.689913 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.690036 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.690160 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] auth.external = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.690281 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.690397 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.690511 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] auth.password = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.690511 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] auth.token = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.690735 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.690858 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.690973 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.691087 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.691201 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] auth.password = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.691201 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] auth.token = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.691426 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.691540 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.691655 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.691769 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.691883 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.692014 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.692131 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.692245 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.692359 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.692473 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.692594 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] credential.caching = True {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.692718 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.692834 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.692948 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.693062 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.693176 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] credential.caching = True {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.693315 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.693446 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.693562 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.693677 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.693800 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.693916 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.694030 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.694155 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.694271 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.694386 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.694500 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.694614 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.694734 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.694858 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.694980 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.695096 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.695210 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.695324 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.695438 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.695561 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.695676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.695797 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.695913 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.696027 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.696140 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.696262 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.696384 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] federation.caching = True {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.696499 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.696612 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.696726 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.696848 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.696972 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] federation.caching = True {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.697087 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.697203 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.697321 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.697422 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.697538 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.697665 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.697782 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.697897 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.698011 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.698125 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.698285 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.698490 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.698735 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.698856 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.698974 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.699088 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.699210 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.699327 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.699450 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] identity.caching = True {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.699564 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.699678 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.699792 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.699972 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.700081 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] identity.caching = True {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.700193 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.700308 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.700421 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.700535 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.700657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.700780 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.700895 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.701009 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.701130 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.701130 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.701407 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.701525 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.701639 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.701752 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.701752 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.701973 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.702098 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.702215 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.702329 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.702443 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.702556 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.702670 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.702792 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.702907 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.703021 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.703136 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.703250 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.703365 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.703365 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.703588 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.703702 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.703816 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.703816 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.704029 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.704151 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.704268 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.704387 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.704501 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.704614 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.704727 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.704849 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.704965 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.705078 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.705192 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.705321 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.705437 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.705563 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.705680 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.705794 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.705907 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.706020 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.706134 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.706134 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.706359 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.706472 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.706586 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.706699 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.706813 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.706936 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.707057 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.707172 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.707285 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.707400 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.707513 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.707635 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.707750 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.707863 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.707977 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.708091 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.708212 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.708328 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.708442 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.708556 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.708669 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.708782 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.708903 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.709019 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.709132 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.709246 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.709381 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.709497 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.709622 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.709738 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.709852 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.709965 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.710087 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.710203 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.710330 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.710446 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.710560 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.710673 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.710787 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.710900 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.711022 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.711137 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.711250 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.user = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.711363 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.711476 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.711591 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.711711 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.711827 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.711940 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.712054 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.user = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.712168 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.712282 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.712403 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.712518 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.712634 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.712747 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.712860 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.712974 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.713095 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.713210 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.713379 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.713494 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.713609 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.713722 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.713869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.713987 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.714101 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.714215 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.714332 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.714446 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.714585 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.714750 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.714870 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.714985 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.715099 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.715213 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.715336 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.715452 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.715565 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.715678 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.715793 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.715907 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.716029 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.716145 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.716260 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.716373 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.716487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.716600 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.716722 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.716838 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.716838 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.717052 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.717052 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.717284 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.717413 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.717529 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.717643 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.717764 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.717880 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.717993 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.718120 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.718236 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.718350 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.718464 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.718578 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.718691 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.718813 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.718930 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.719044 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.719158 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.719272 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.719385 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.719507 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.719623 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.719737 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.719850 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.719965 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] resource.caching = True {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.720070 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.720192 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.720307 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.720420 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.720534 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.720648 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.720762 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] resource.caching = True {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.720884 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.721001 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.721115 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.721228 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.721369 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.721487 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.721609 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.721725 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.721839 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.721952 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.722067 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.722180 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.722366 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] role.caching = True {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.722484 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.722598 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] role.driver = sql {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.722712 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.722826 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.722948 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.723066 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.723180 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] role.caching = True {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.723293 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.723408 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] role.driver = sql {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.723521 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.723644 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.723759 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.723873 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.723987 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.724101 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.724218 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.724341 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.724457 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.724570 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.724684 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.724798 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.724912 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.725033 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.725150 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.725293 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.725410 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.725525 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.725638 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.725763 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.725879 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.725993 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.725993 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.726208 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.726321 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.726443 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.726559 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.726672 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.726672 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.726886 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.727000 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.727122 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.727238 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.727352 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.727466 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.727580 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.727694 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.727816 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.727816 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.727816 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.728131 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.728245 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.728367 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.728483 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.728596 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.728710 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.728824 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.728938 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.729060 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.729175 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.729307 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.729945 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.730069 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.730223 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.730402 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.730524 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.730639 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.730794 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.730953 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.731070 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.731193 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.731309 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.731424 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] token.caching = True {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.731545 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.731660 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.731774 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] token.caching = True {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.731896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.732017 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.732132 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.732246 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.732359 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.732474 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.732596 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.732712 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.732865 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.732865 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.733090 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.733204 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.733392 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.733510 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.733510 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.733727 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.733841 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.733955 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.734078 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.734184 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.734297 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.734411 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.734524 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.734639 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.734764 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.734890 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.735014 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.735130 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.735244 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.735358 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.735472 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] database.connection = **** {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.735594 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.735719 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.735847 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.735962 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.736076 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.736190 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.736303 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] database.connection = **** {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.736417 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.736539 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.736655 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.736770 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.736883 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.736997 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.737111 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.737233 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.737367 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.737481 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.737845 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.737961 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.738075 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.738202 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.738318 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.738433 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.738547 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.738661 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.738775 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.738898 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.739013 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.739127 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.739241 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.739355 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.739469 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.739591 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.739707 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.739823 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.739937 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.740051 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.740165 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81387) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Sep 25 00:40:30.740289 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.740289 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.740289 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 00:40:30.740289 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81386) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Sep 25 00:40:30.766949 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81387) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:30.767179 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81387) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:30.767406 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81387) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:30.767490 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81386) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:30.767696 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81386) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:30.767914 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81386) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:30.776099 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81387) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:30.776193 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81386) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:30.805868 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81386) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:30.806033 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81387) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:30.806110 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81386) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:30.806185 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81387) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:30.860229 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81386) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:30.860393 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81387) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:30.908900 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81386) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:30.909119 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81386) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:30.909266 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81387) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:30.909508 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81387) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:31.068226 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81386) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:31.068396 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81386) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:31.068581 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81386) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:31.068795 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81386) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:31.069006 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81386) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:31.069214 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81386) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:31.072661 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81387) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:31.072894 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81387) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:31.073117 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81387) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:31.073389 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81387) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:31.073610 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81387) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:31.073825 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81387) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:31.092715 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81387) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:31.092983 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81387) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:31.093202 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81387) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:31.093451 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81387) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:31.093672 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81387) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:31.093754 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81386) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:31.093890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81387) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:31.093974 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81386) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:31.094107 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81387) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:31.094193 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81386) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:31.094326 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81386) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:31.094543 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81386) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:31.094775 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81386) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:31.094989 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81386) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:31.106045 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81387) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:31.107015 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81386) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:31.119040 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81387) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:31.119272 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81387) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:31.119442 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81386) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:31.119545 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81387) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:31.119669 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81386) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:31.119752 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81387) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:31.119889 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81386) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:31.119971 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81387) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:31.120148 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81386) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:31.120235 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81387) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:31.120454 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81386) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:31.120582 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81387) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:31.120817 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81386) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:31.121149 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81386) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 00:40:31.137731 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.137939 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 25 00:40:31.138281 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.138483 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 25 00:40:31.138760 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.138898 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.139025 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.139107 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 25 00:40:31.139286 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.139424 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.139546 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.139629 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 25 00:40:31.139872 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.140125 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.140377 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.140629 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.149129 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.149435 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.149709 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.149996 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.150326 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.150603 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.150870 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.151150 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.157666 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81387) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 00:40:31.159139 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81386) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 00:40:31.160575 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81387) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 00:40:31.160957 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.161144 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.161459 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.161744 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.162009 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.162104 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81386) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 00:40:31.162282 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.162364 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.162558 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.162642 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.162831 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.162914 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.163107 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.163189 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.163379 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.163468 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.163651 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.164003 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.164204 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.164497 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.164763 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.186740 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.188211 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81387) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 00:40:31.188497 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.188585 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.190084 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81386) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 00:40:31.190364 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.193224 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81387) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 00:40:31.193533 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.193819 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.195155 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81386) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 00:40:31.195437 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.195714 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.200899 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81387) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 00:40:31.201184 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.203079 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81386) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 00:40:31.203361 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.206920 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81387) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 00:40:31.207220 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.207512 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.207800 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.208078 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.208352 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.208627 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.208897 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.209144 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81386) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 00:40:31.209487 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.209783 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.210064 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.210335 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.210608 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.210894 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.211160 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.225370 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.225651 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.228568 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.228865 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.229632 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81387) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 00:40:31.233152 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.233485 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.233669 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81386) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 00:40:31.236959 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.237279 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.237971 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81387) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 00:40:31.241082 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81387) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 00:40:31.241848 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81386) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 00:40:31.244357 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.244656 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.244941 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.245061 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81386) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 00:40:31.245294 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.245584 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.245891 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.248679 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.248974 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.249286 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.249562 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.249842 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.250130 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.262875 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81387) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 00:40:31.263182 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.263457 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.263726 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.267014 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81386) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 00:40:31.267313 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.267579 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.267839 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.270743 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.272317 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.273889 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.274169 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.274773 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.276355 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.277549 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81387) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 00:40:31.277925 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.278251 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.278542 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.278776 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.278937 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.279049 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.281832 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81386) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 00:40:31.282115 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.282395 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.282680 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.283065 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.290198 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81387) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 00:40:31.290489 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.290773 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.291056 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.291438 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.291732 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.292126 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.294697 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81386) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 00:40:31.294979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.295255 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.295531 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.295906 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.296183 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.296467 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.307843 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81387) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 00:40:31.310799 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81387) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 00:40:31.312384 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81386) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 00:40:31.313730 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.315261 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.315344 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81386) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 00:40:31.316748 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81387) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 00:40:31.317032 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.317454 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.318212 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.319694 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.321104 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81386) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 00:40:31.321418 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.321821 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.324388 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.325881 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81387) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 00:40:31.328788 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.329090 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.329175 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.329485 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.329767 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.330569 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81386) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 00:40:31.333469 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.333868 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.334148 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.334421 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.337939 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81387) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 00:40:31.338238 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.338522 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.343116 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81386) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 00:40:31.343402 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.343681 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.346163 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81387) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 00:40:31.346453 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.346738 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.347019 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.347295 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.347568 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.347845 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.348143 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.348417 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.348687 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.348968 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.349237 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.349535 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.349815 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81387) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.350575 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81386) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 00:40:31.350851 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.351126 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.351395 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.351663 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.351929 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.352197 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.352475 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.352740 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.353016 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.353306 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.353571 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.353829 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.354144 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81386) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 00:40:31.384673 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x562cf5a699c0 pid: 81387 (default app) Sep 25 00:40:31.388617 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54604e96-1e28-4f0d-ae4f-df43f1b65a86 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:31.388905 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54604e96-1e28-4f0d-ae4f-df43f1b65a86 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:31.389178 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54604e96-1e28-4f0d-ae4f-df43f1b65a86 None None] PATH_INFO: `/v3` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:31.390017 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x562cf5a699c0 pid: 81386 (default app) Sep 25 00:40:31.393358 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1/1] 10.222.0.52 () {50 vars in 791 bytes} [Sun Sep 25 00:40:31 2022] GET /identity/v3/ => generated 255 bytes in 8 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:40:38.344639 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613a1814-49b3-41da-84d6-d1a6f5b40ee5 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:38.345170 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613a1814-49b3-41da-84d6-d1a6f5b40ee5 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:38.345170 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613a1814-49b3-41da-84d6-d1a6f5b40ee5 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:38.349442 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1/2] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:40:38 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:40:38.354742 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54778ef3-a20a-455a-8c70-5fe763ca1e05 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:38.355273 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54778ef3-a20a-455a-8c70-5fe763ca1e05 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:38.355273 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54778ef3-a20a-455a-8c70-5fe763ca1e05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:38.426725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: INFO dbcounter [None req-54778ef3-a20a-455a-8c70-5fe763ca1e05 None None] Registered counter for database keystone Sep 25 00:40:38.446190 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG oslo_db.sqlalchemy.engines [None req-54778ef3-a20a-455a-8c70-5fe763ca1e05 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81387) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Sep 25 00:40:38.453347 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writer thread running {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Sep 25 00:40:38.636865 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG passlib.handlers.bcrypt [None req-54778ef3-a20a-455a-8c70-5fe763ca1e05 None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81387) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Sep 25 00:40:38.637159 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG passlib.handlers.bcrypt [None req-54778ef3-a20a-455a-8c70-5fe763ca1e05 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81387) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Sep 25 00:40:38.696241 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-54778ef3-a20a-455a-8c70-5fe763ca1e05 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:40:38.697901 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-54778ef3-a20a-455a-8c70-5fe763ca1e05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:40:38.835015 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 2/3] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:40:38 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 481 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Sep 25 00:40:39.319125 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-613a1814-49b3-41da-84d6-d1a6f5b40ee5 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:40:39.390367 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: INFO dbcounter [None req-613a1814-49b3-41da-84d6-d1a6f5b40ee5 None None] Registered counter for database keystone Sep 25 00:40:39.412326 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG oslo_db.sqlalchemy.engines [None req-613a1814-49b3-41da-84d6-d1a6f5b40ee5 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81386) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Sep 25 00:40:39.419456 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writer thread running {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Sep 25 00:40:39.538987 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-613a1814-49b3-41da-84d6-d1a6f5b40ee5 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:40:39.544177 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-390d750c-21df-4210-8c11-6b170328a9aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:40:39.544804 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390d750c-21df-4210-8c11-6b170328a9aa admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:39.545076 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390d750c-21df-4210-8c11-6b170328a9aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:39.545366 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390d750c-21df-4210-8c11-6b170328a9aa admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:39.553703 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-390d750c-21df-4210-8c11-6b170328a9aa admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:40:39.606385 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: WARNING keystone.common.rbac_enforcer.enforcer [None req-390d750c-21df-4210-8c11-6b170328a9aa admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Sep 25 00:40:39.606634 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-390d750c-21df-4210-8c11-6b170328a9aa admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:40:39.617163 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 2/4] 10.222.0.52 () {58 vars in 1324 bytes} [Sun Sep 25 00:40:39 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 299 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 25 00:40:39.624174 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54778ef3-a20a-455a-8c70-5fe763ca1e05 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:40:39.642373 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54778ef3-a20a-455a-8c70-5fe763ca1e05 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:40:39.646742 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b1a7f29-95ab-48d0-b6a0-86236ce44e0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:40:39.647866 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1a7f29-95ab-48d0-b6a0-86236ce44e0c admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:39.648142 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1a7f29-95ab-48d0-b6a0-86236ce44e0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:39.648409 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1a7f29-95ab-48d0-b6a0-86236ce44e0c admin admin] PATH_INFO: `/v3/projects/c8e32a9668254c2e96354fd797883364` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:39.649998 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b1a7f29-95ab-48d0-b6a0-86236ce44e0c admin admin] RBAC: Authorizing `identity:get_project(project_id=c8e32a9668254c2e96354fd797883364)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:40:39.703832 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: WARNING keystone.common.rbac_enforcer.enforcer [None req-8b1a7f29-95ab-48d0-b6a0-86236ce44e0c admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Sep 25 00:40:39.704169 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b1a7f29-95ab-48d0-b6a0-86236ce44e0c admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:40:39.708818 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 3/5] 10.222.0.52 () {56 vars in 1207 bytes} [Sun Sep 25 00:40:39 2022] GET /identity/v3/projects/c8e32a9668254c2e96354fd797883364 => generated 344 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:40:41.907069 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5f5ec4-d743-4e83-93d6-177d5db12ec2 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:41.907603 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5f5ec4-d743-4e83-93d6-177d5db12ec2 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:41.907603 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5f5ec4-d743-4e83-93d6-177d5db12ec2 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:41.911689 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 3/6] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:40:41 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 00:40:41.917325 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f339cf4d-c418-4f1e-b954-b939d4659b50 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:41.917750 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f339cf4d-c418-4f1e-b954-b939d4659b50 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:41.917871 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f339cf4d-c418-4f1e-b954-b939d4659b50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:41.937917 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-f339cf4d-c418-4f1e-b954-b939d4659b50 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:40:41.939420 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-f339cf4d-c418-4f1e-b954-b939d4659b50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:40:41.947291 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 4/7] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:40:41 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Sep 25 00:40:42.220113 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a5f5ec4-d743-4e83-93d6-177d5db12ec2 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:40:42.232675 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a5f5ec4-d743-4e83-93d6-177d5db12ec2 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:40:42.236835 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75eeaa3a-198e-4eb3-94bc-74e9796e8cf8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:40:42.237468 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75eeaa3a-198e-4eb3-94bc-74e9796e8cf8 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:42.238007 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75eeaa3a-198e-4eb3-94bc-74e9796e8cf8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:42.238270 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75eeaa3a-198e-4eb3-94bc-74e9796e8cf8 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:42.247200 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75eeaa3a-198e-4eb3-94bc-74e9796e8cf8 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:40:42.248971 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75eeaa3a-198e-4eb3-94bc-74e9796e8cf8 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:40:42.257740 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 4/8] 10.222.0.52 () {58 vars in 1324 bytes} [Sun Sep 25 00:40:42 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 25 00:40:42.264470 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f339cf4d-c418-4f1e-b954-b939d4659b50 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:40:42.273660 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f339cf4d-c418-4f1e-b954-b939d4659b50 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:40:42.278111 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54712975-20ea-4d93-9ac0-99aa20dd30bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:40:42.278739 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54712975-20ea-4d93-9ac0-99aa20dd30bd admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:42.279013 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54712975-20ea-4d93-9ac0-99aa20dd30bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:42.279282 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54712975-20ea-4d93-9ac0-99aa20dd30bd admin admin] PATH_INFO: `/v3/users/d6d73e23626043529ac5b9e3889933d4` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:42.280587 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54712975-20ea-4d93-9ac0-99aa20dd30bd admin admin] RBAC: Authorizing `identity:get_user(user_id=d6d73e23626043529ac5b9e3889933d4)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:40:42.282430 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54712975-20ea-4d93-9ac0-99aa20dd30bd admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:40:42.283964 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 5/9] 10.222.0.52 () {56 vars in 1198 bytes} [Sun Sep 25 00:40:42 2022] GET /identity/v3/users/d6d73e23626043529ac5b9e3889933d4 => generated 247 bytes in 20 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:40:44.575681 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30205e1e-2898-4531-be01-0068a66aea98 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:44.576220 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30205e1e-2898-4531-be01-0068a66aea98 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:44.576220 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30205e1e-2898-4531-be01-0068a66aea98 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:44.577552 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 5/10] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:40:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:40:44.581024 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb5c2e0-57d9-486c-b42a-610c496e45cf None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:44.581514 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb5c2e0-57d9-486c-b42a-610c496e45cf None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:44.581692 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb5c2e0-57d9-486c-b42a-610c496e45cf None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:44.582912 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 6/11] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:40:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:40:44.583110 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669f6802-ec40-43ab-94aa-088adbb1b885 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:44.583346 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669f6802-ec40-43ab-94aa-088adbb1b885 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:44.583619 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669f6802-ec40-43ab-94aa-088adbb1b885 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:44.588203 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e11d00-872b-44bc-9154-c60433be456c None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:44.589007 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e11d00-872b-44bc-9154-c60433be456c None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:44.589314 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e11d00-872b-44bc-9154-c60433be456c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:44.607881 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-17e11d00-872b-44bc-9154-c60433be456c None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:40:44.609066 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-17e11d00-872b-44bc-9154-c60433be456c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:40:44.627516 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG passlib.handlers.bcrypt [None req-669f6802-ec40-43ab-94aa-088adbb1b885 None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81386) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Sep 25 00:40:44.627905 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG passlib.handlers.bcrypt [None req-669f6802-ec40-43ab-94aa-088adbb1b885 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81386) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Sep 25 00:40:44.677352 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-669f6802-ec40-43ab-94aa-088adbb1b885 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:40:44.678986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-669f6802-ec40-43ab-94aa-088adbb1b885 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:40:44.691850 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 7/12] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:40:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 25 00:40:44.780880 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 6/13] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:40:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:40:44.962474 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e11d00-872b-44bc-9154-c60433be456c None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:40:44.987647 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:40:45.049949 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-669f6802-ec40-43ab-94aa-088adbb1b885 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:40:45.076276 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e11d00-872b-44bc-9154-c60433be456c None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:40:45.116670 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e91c90c-a9d2-42cb-84f8-e41af7a9869c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:40:45.117322 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e91c90c-a9d2-42cb-84f8-e41af7a9869c None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:45.117605 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e91c90c-a9d2-42cb-84f8-e41af7a9869c None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:45.117880 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e91c90c-a9d2-42cb-84f8-e41af7a9869c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:45.156456 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-669f6802-ec40-43ab-94aa-088adbb1b885 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:40:45.158825 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:40:45.190706 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e91c90c-a9d2-42cb-84f8-e41af7a9869c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:40:45.192191 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e91c90c-a9d2-42cb-84f8-e41af7a9869c None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:40:45.196525 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-464aa543-758f-45c9-8af0-39ec38c272ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:40:45.197677 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464aa543-758f-45c9-8af0-39ec38c272ba None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:45.198021 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464aa543-758f-45c9-8af0-39ec38c272ba None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:45.198410 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464aa543-758f-45c9-8af0-39ec38c272ba None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:45.210666 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-464aa543-758f-45c9-8af0-39ec38c272ba None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:40:45.216561 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:40:45.220322 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-464aa543-758f-45c9-8af0-39ec38c272ba None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:40:45.220322 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:40:45.220322 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:40:45.220322 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:40:45.220322 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:40:45.220322 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:40:45.220322 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:40:45.220322 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:40:45.220322 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:40:45.220322 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:40:45.220322 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:40:45.220322 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:40:45.220322 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:40:45.220322 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:40:45.220322 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:40:45.220322 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:40:45.221402 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:40:45.221402 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:40:45.221402 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:40:45.221402 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:40:45.221402 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:40:45.221402 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:40:45.221402 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:40:45.221402 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:40:45.221402 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:40:45.221402 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:40:45.221402 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:40:45.221402 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:40:45.221402 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:40:45.221402 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:40:45.221402 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:40:45.221402 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:40:45.222234 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:40:45.222234 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:40:45.222234 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:40:45.222234 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:40:45.222234 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:40:45.222234 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:40:45.222234 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:40:45.222234 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:40:45.222234 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:40:45.222234 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:40:45.222234 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:40:45.222234 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:40:45.222234 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 7/14] 10.222.0.52 () {56 vars in 1096 bytes} [Sun Sep 25 00:40:45 2022] GET /identity/v3/roles/admin => generated 83 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:40:45.227411 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-464aa543-758f-45c9-8af0-39ec38c272ba None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:40:45.298767 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 8/15] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:40:44 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 337 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 00:40:45.304529 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e91c90c-a9d2-42cb-84f8-e41af7a9869c None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:40:45.331080 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-464aa543-758f-45c9-8af0-39ec38c272ba None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:40:45.366185 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9907ef29-833f-4c12-8027-2e71adc9e433 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:40:45.366783 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9907ef29-833f-4c12-8027-2e71adc9e433 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:45.367064 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9907ef29-833f-4c12-8027-2e71adc9e433 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:45.367356 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9907ef29-833f-4c12-8027-2e71adc9e433 None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:45.368019 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9907ef29-833f-4c12-8027-2e71adc9e433 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:40:45.368317 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9907ef29-833f-4c12-8027-2e71adc9e433 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:40:45.369885 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9907ef29-833f-4c12-8027-2e71adc9e433 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:40:45.383089 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 8/16] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:40:45 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:40:45.389056 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9907ef29-833f-4c12-8027-2e71adc9e433 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:40:45.407067 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e91c90c-a9d2-42cb-84f8-e41af7a9869c None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:40:45.416072 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:40:45.429923 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:40:45.441921 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b07b5b07-5444-44af-b9c1-c8cb3935fbfa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:40:45.442547 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07b5b07-5444-44af-b9c1-c8cb3935fbfa None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:45.442821 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07b5b07-5444-44af-b9c1-c8cb3935fbfa None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:45.443088 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07b5b07-5444-44af-b9c1-c8cb3935fbfa None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:45.454317 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b07b5b07-5444-44af-b9c1-c8cb3935fbfa None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:40:45.455789 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b07b5b07-5444-44af-b9c1-c8cb3935fbfa None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:40:45.475092 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-b07b5b07-5444-44af-b9c1-c8cb3935fbfa None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:40:45.475092 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:40:45.475092 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:40:45.475092 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:40:45.475092 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:40:45.475092 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:40:45.475092 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:40:45.475092 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:40:45.475092 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:40:45.475092 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:40:45.475092 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:40:45.475092 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:40:45.475092 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:40:45.475092 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:40:45.475092 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:40:45.475092 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:40:45.476043 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:40:45.476043 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:40:45.476043 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:40:45.476043 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:40:45.476043 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:40:45.476043 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:40:45.476043 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:40:45.476043 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:40:45.476043 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:40:45.476043 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:40:45.476043 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:40:45.476043 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:40:45.476043 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:40:45.476043 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:40:45.476043 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:40:45.476043 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:40:45.476043 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:40:45.476861 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:40:45.476861 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:40:45.476861 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:40:45.476861 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 9/17] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:40:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:40:45.482607 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b07b5b07-5444-44af-b9c1-c8cb3935fbfa None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:40:45.496074 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9907ef29-833f-4c12-8027-2e71adc9e433 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:40:45.543632 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e420e0a5-1a02-4128-b778-5bb73dac523f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:40:45.544265 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e420e0a5-1a02-4128-b778-5bb73dac523f None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:45.544538 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e420e0a5-1a02-4128-b778-5bb73dac523f None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:45.544812 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e420e0a5-1a02-4128-b778-5bb73dac523f None admin] PATH_INFO: `/v3/users/d6d73e23626043529ac5b9e3889933d4` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:45.546316 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e420e0a5-1a02-4128-b778-5bb73dac523f None admin] RBAC: Authorizing `identity:get_user(user_id=d6d73e23626043529ac5b9e3889933d4)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:40:45.547755 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e420e0a5-1a02-4128-b778-5bb73dac523f None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:40:45.549357 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 9/18] 10.222.0.52 () {56 vars in 1177 bytes} [Sun Sep 25 00:40:45 2022] GET /identity/v3/users/d6d73e23626043529ac5b9e3889933d4 => generated 247 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:40:45.554990 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e420e0a5-1a02-4128-b778-5bb73dac523f None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:40:45.589897 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b07b5b07-5444-44af-b9c1-c8cb3935fbfa None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:40:45.634290 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e8cd62-e38c-44f9-a68d-70a91435f08e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:40:45.634897 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e8cd62-e38c-44f9-a68d-70a91435f08e None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:45.635181 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e8cd62-e38c-44f9-a68d-70a91435f08e None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:45.635576 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e8cd62-e38c-44f9-a68d-70a91435f08e None admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:45.636250 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57e8cd62-e38c-44f9-a68d-70a91435f08e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:40:45.636561 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57e8cd62-e38c-44f9-a68d-70a91435f08e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:40:45.637983 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57e8cd62-e38c-44f9-a68d-70a91435f08e None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:40:45.652042 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:40:45.656488 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 10/19] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:40:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:40:45.660268 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e420e0a5-1a02-4128-b778-5bb73dac523f None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:40:45.699423 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee52302a-a7c7-45f9-a763-f31e0bc158d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:40:45.700058 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee52302a-a7c7-45f9-a763-f31e0bc158d0 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:45.700328 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee52302a-a7c7-45f9-a763-f31e0bc158d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:45.700591 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee52302a-a7c7-45f9-a763-f31e0bc158d0 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:45.701772 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee52302a-a7c7-45f9-a763-f31e0bc158d0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:40:45.703224 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee52302a-a7c7-45f9-a763-f31e0bc158d0 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:40:45.704807 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 10/20] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:40:45 2022] GET /identity/v3/domains/default => generated 199 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:40:45.711088 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e8cd62-e38c-44f9-a68d-70a91435f08e None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:40:45.750987 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:40:45.824769 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e8cd62-e38c-44f9-a68d-70a91435f08e None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:40:45.862161 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779e2e96-44de-4b7f-a035-3afde83523e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:40:45.863033 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779e2e96-44de-4b7f-a035-3afde83523e5 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:45.863311 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779e2e96-44de-4b7f-a035-3afde83523e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:45.863577 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779e2e96-44de-4b7f-a035-3afde83523e5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:45.864306 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-779e2e96-44de-4b7f-a035-3afde83523e5 None admin] RBAC: Adding query filter params (role.id=2c242abf6ed8410ab0a4a354ca1794fb, scope.domain.id=default, user.id=d6d73e23626043529ac5b9e3889933d4) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:40:45.864612 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-779e2e96-44de-4b7f-a035-3afde83523e5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:40:45.866143 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-779e2e96-44de-4b7f-a035-3afde83523e5 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:40:45.873399 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 11/21] 10.222.0.52 () {56 vars in 1322 bytes} [Sun Sep 25 00:40:45 2022] GET /identity/v3/role_assignments?user.id=d6d73e23626043529ac5b9e3889933d4&scope.domain.id=default&role.id=2c242abf6ed8410ab0a4a354ca1794fb => generated 233 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:40:48.290121 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded088c7-3d98-43d8-9a11-257f6af28a74 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:48.291151 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded088c7-3d98-43d8-9a11-257f6af28a74 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:48.291891 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded088c7-3d98-43d8-9a11-257f6af28a74 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:48.293879 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 11/22] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:40:48 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:40:48.315461 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63eca773-f7b3-4797-9eb1-0eb0a3105b96 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:48.316462 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63eca773-f7b3-4797-9eb1-0eb0a3105b96 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:48.316939 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63eca773-f7b3-4797-9eb1-0eb0a3105b96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:48.346293 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-63eca773-f7b3-4797-9eb1-0eb0a3105b96 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:40:48.347923 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-63eca773-f7b3-4797-9eb1-0eb0a3105b96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:40:48.445471 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 12/23] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:40:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:40:48.458503 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee81959-26c2-4da3-9abb-db2e32bc23ab None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:48.459148 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee81959-26c2-4da3-9abb-db2e32bc23ab None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:48.459618 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee81959-26c2-4da3-9abb-db2e32bc23ab None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:48.461293 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 12/24] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:40:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:40:48.468190 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0165b508-369b-448f-9bec-c81e72508586 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:48.468838 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0165b508-369b-448f-9bec-c81e72508586 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:48.469362 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0165b508-369b-448f-9bec-c81e72508586 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:48.501800 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-0165b508-369b-448f-9bec-c81e72508586 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:40:48.503335 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-0165b508-369b-448f-9bec-c81e72508586 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:40:48.529967 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:40:48.578831 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 13/25] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:40:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:40:48.654356 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91741e69-9471-40e9-821e-f3ea0e6a67cb None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:48.654695 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91741e69-9471-40e9-821e-f3ea0e6a67cb None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:48.654899 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91741e69-9471-40e9-821e-f3ea0e6a67cb None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:48.656307 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 13/26] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:40:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:40:48.662081 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0efc413b-37b5-49ae-b78f-6b902d5a522c None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:48.662362 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0efc413b-37b5-49ae-b78f-6b902d5a522c None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:48.662632 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0efc413b-37b5-49ae-b78f-6b902d5a522c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:48.686362 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-0efc413b-37b5-49ae-b78f-6b902d5a522c None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:40:48.687565 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-0efc413b-37b5-49ae-b78f-6b902d5a522c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:40:48.791777 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 14/27] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:40:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:40:48.863200 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91741e69-9471-40e9-821e-f3ea0e6a67cb None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:40:48.954781 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0efc413b-37b5-49ae-b78f-6b902d5a522c None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:40:48.989946 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91741e69-9471-40e9-821e-f3ea0e6a67cb None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:40:49.028164 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ea49285-1600-4ae1-93d8-cd4c41918aac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:40:49.029656 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea49285-1600-4ae1-93d8-cd4c41918aac None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:49.030140 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea49285-1600-4ae1-93d8-cd4c41918aac None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:49.030617 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea49285-1600-4ae1-93d8-cd4c41918aac None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:49.052197 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ea49285-1600-4ae1-93d8-cd4c41918aac None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:40:49.054111 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ea49285-1600-4ae1-93d8-cd4c41918aac None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:40:49.072058 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-7ea49285-1600-4ae1-93d8-cd4c41918aac None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:40:49.072058 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:40:49.072058 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:40:49.072058 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:40:49.072058 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:40:49.072058 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:40:49.072058 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:40:49.072058 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:40:49.072058 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:40:49.072058 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:40:49.072058 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:40:49.072058 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:40:49.072058 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:40:49.072058 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:40:49.072058 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:40:49.072058 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:40:49.073032 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:40:49.073032 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:40:49.073032 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:40:49.073032 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:40:49.073032 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:40:49.073032 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:40:49.073032 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:40:49.073032 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:40:49.073032 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:40:49.073032 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:40:49.073032 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:40:49.073032 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:40:49.073032 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:40:49.073032 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:40:49.073032 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:40:49.073032 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:40:49.073032 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:40:49.073899 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:40:49.073899 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:40:49.073899 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:40:49.076105 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 14/28] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:40:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:40:49.085064 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ea49285-1600-4ae1-93d8-cd4c41918aac None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:40:49.109646 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:40:49.201163 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ea49285-1600-4ae1-93d8-cd4c41918aac None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:40:49.239864 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5304393-c983-4af6-a77d-b04a06dbc89f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:40:49.240882 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5304393-c983-4af6-a77d-b04a06dbc89f None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:49.241427 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5304393-c983-4af6-a77d-b04a06dbc89f None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:49.241929 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5304393-c983-4af6-a77d-b04a06dbc89f None admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:49.242785 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5304393-c983-4af6-a77d-b04a06dbc89f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:40:49.243282 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5304393-c983-4af6-a77d-b04a06dbc89f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:40:49.244898 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5304393-c983-4af6-a77d-b04a06dbc89f None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:40:49.252464 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:40:49.268089 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 15/29] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:40:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:40:49.276672 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5304393-c983-4af6-a77d-b04a06dbc89f None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:40:49.282954 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0efc413b-37b5-49ae-b78f-6b902d5a522c None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:40:49.356720 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9498788-2864-4cd0-bffc-413b8bfac8c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:40:49.357544 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9498788-2864-4cd0-bffc-413b8bfac8c5 None admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:49.357647 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9498788-2864-4cd0-bffc-413b8bfac8c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:49.357894 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9498788-2864-4cd0-bffc-413b8bfac8c5 None admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:49.358599 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9498788-2864-4cd0-bffc-413b8bfac8c5 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:40:49.360002 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9498788-2864-4cd0-bffc-413b8bfac8c5 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:40:49.377614 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:40:49.409990 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5304393-c983-4af6-a77d-b04a06dbc89f None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:40:49.450476 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9063a97-c4ea-4a45-90b6-ea0ee8749bef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:40:49.451625 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9063a97-c4ea-4a45-90b6-ea0ee8749bef None admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:49.452109 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9063a97-c4ea-4a45-90b6-ea0ee8749bef None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:49.452603 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9063a97-c4ea-4a45-90b6-ea0ee8749bef None admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:49.453529 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9063a97-c4ea-4a45-90b6-ea0ee8749bef None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:40:49.455217 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9063a97-c4ea-4a45-90b6-ea0ee8749bef None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:40:49.561436 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 15/30] 10.222.0.52 () {60 vars in 1131 bytes} [Sun Sep 25 00:40:48 2022] POST /identity/v3/roles => generated 219 bytes in 600 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 00:40:49.562678 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9498788-2864-4cd0-bffc-413b8bfac8c5 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:40:49.626480 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 16/31] 10.222.0.52 () {60 vars in 1141 bytes} [Sun Sep 25 00:40:49 2022] POST /identity/v3/projects => generated 301 bytes in 350 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:40:49.628482 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7316cb1-77ad-4be9-9346-9341c666f1d2 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:49.629070 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7316cb1-77ad-4be9-9346-9341c666f1d2 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:49.629667 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7316cb1-77ad-4be9-9346-9341c666f1d2 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:49.631377 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 17/32] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:40:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:40:49.645516 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74cf77a-adc0-4a5d-a783-4fd874d99093 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:49.646431 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74cf77a-adc0-4a5d-a783-4fd874d99093 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:49.646916 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74cf77a-adc0-4a5d-a783-4fd874d99093 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:49.682547 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-a74cf77a-adc0-4a5d-a783-4fd874d99093 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:40:49.684917 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-a74cf77a-adc0-4a5d-a783-4fd874d99093 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:40:49.829605 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9498788-2864-4cd0-bffc-413b8bfac8c5 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:40:49.844870 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 18/33] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:40:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:40:49.884433 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f0338b3-c425-4ef9-8fe0-3aa1662e360b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:40:49.886315 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f0338b3-c425-4ef9-8fe0-3aa1662e360b None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:49.886911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f0338b3-c425-4ef9-8fe0-3aa1662e360b None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:49.887392 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f0338b3-c425-4ef9-8fe0-3aa1662e360b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:49.904162 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f0338b3-c425-4ef9-8fe0-3aa1662e360b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:40:49.913306 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-1f0338b3-c425-4ef9-8fe0-3aa1662e360b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:40:49.913306 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:40:49.913306 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:40:49.913306 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:40:49.913306 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:40:49.913306 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:40:49.913306 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:40:49.913306 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:40:49.913306 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:40:49.913306 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:40:49.913306 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:40:49.913306 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:40:49.913306 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:40:49.913306 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:40:49.913306 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:40:49.913306 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:40:49.914221 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:40:49.914221 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:40:49.914221 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:40:49.914221 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:40:49.914221 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:40:49.914221 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:40:49.914221 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:40:49.914221 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:40:49.914221 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:40:49.914221 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:40:49.914221 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:40:49.914221 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:40:49.914221 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:40:49.914221 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:40:49.914221 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:40:49.914221 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:40:49.914997 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:40:49.914997 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:40:49.914997 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:40:49.914997 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:40:49.914997 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:40:49.914997 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:40:49.914997 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:40:49.914997 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:40:49.914997 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:40:49.914997 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:40:49.914997 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:40:49.914997 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:40:49.918449 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 16/34] 10.222.0.52 () {56 vars in 1096 bytes} [Sun Sep 25 00:40:49 2022] GET /identity/v3/roles/admin => generated 83 bytes in 356 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:40:49.926532 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a74cf77a-adc0-4a5d-a783-4fd874d99093 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:40:50.019248 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:40:50.079856 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a74cf77a-adc0-4a5d-a783-4fd874d99093 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:40:50.128366 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca99db8-4f49-40cb-94bb-30bb4b424787 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:50.128563 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f7c3c4f-96d9-4802-a0e2-19030e70e63c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:40:50.129072 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7c3c4f-96d9-4802-a0e2-19030e70e63c None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:50.129376 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7c3c4f-96d9-4802-a0e2-19030e70e63c None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:50.129652 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7c3c4f-96d9-4802-a0e2-19030e70e63c None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:50.130066 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca99db8-4f49-40cb-94bb-30bb4b424787 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:50.130339 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f7c3c4f-96d9-4802-a0e2-19030e70e63c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:40:50.130642 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f7c3c4f-96d9-4802-a0e2-19030e70e63c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:40:50.131053 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca99db8-4f49-40cb-94bb-30bb4b424787 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:50.132181 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f7c3c4f-96d9-4802-a0e2-19030e70e63c None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:40:50.133215 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 17/35] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:40:50 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:40:50.144337 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 19/36] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:40:49 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:40:50.146713 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80819763-e828-4fcf-9798-f892c357257b None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:50.147068 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80819763-e828-4fcf-9798-f892c357257b None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:50.148009 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80819763-e828-4fcf-9798-f892c357257b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:50.150130 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f7c3c4f-96d9-4802-a0e2-19030e70e63c None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:40:50.185115 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-80819763-e828-4fcf-9798-f892c357257b None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:40:50.186824 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-80819763-e828-4fcf-9798-f892c357257b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:40:50.189180 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:40:50.296894 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f7c3c4f-96d9-4802-a0e2-19030e70e63c None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:40:50.306789 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 18/37] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:40:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:40:50.313691 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce2e457-2b13-4ef1-b105-3c970eb3e128 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:50.314237 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce2e457-2b13-4ef1-b105-3c970eb3e128 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:50.314704 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce2e457-2b13-4ef1-b105-3c970eb3e128 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:50.316520 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 19/38] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:40:50 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 00:40:50.322367 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9542bb4-9bee-42ea-b645-59202c4c5e6e None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:50.323650 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9542bb4-9bee-42ea-b645-59202c4c5e6e None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:50.324100 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9542bb4-9bee-42ea-b645-59202c4c5e6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:50.324973 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:40:50.344633 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96fd31a6-bf27-4bea-b14f-949de4cbcb44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:40:50.345293 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96fd31a6-bf27-4bea-b14f-949de4cbcb44 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:50.345575 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96fd31a6-bf27-4bea-b14f-949de4cbcb44 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:50.345843 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96fd31a6-bf27-4bea-b14f-949de4cbcb44 None admin] PATH_INFO: `/v3/users/d6d73e23626043529ac5b9e3889933d4` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:50.345963 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-b9542bb4-9bee-42ea-b645-59202c4c5e6e None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:40:50.346976 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96fd31a6-bf27-4bea-b14f-949de4cbcb44 None admin] RBAC: Authorizing `identity:get_user(user_id=d6d73e23626043529ac5b9e3889933d4)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:40:50.347332 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-b9542bb4-9bee-42ea-b645-59202c4c5e6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:40:50.348404 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96fd31a6-bf27-4bea-b14f-949de4cbcb44 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:40:50.350482 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 20/39] 10.222.0.52 () {56 vars in 1177 bytes} [Sun Sep 25 00:40:50 2022] GET /identity/v3/users/d6d73e23626043529ac5b9e3889933d4 => generated 247 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:40:50.351030 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96fd31a6-bf27-4bea-b14f-949de4cbcb44 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:40:50.423606 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 20/40] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:40:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 25 00:40:50.424315 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9542bb4-9bee-42ea-b645-59202c4c5e6e None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:40:50.471287 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96fd31a6-bf27-4bea-b14f-949de4cbcb44 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:40:50.508043 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e9274f-b215-48d0-990c-adb0538f5fce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:40:50.508662 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:40:50.509279 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e9274f-b215-48d0-990c-adb0538f5fce None admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:50.509815 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e9274f-b215-48d0-990c-adb0538f5fce None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:50.510416 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e9274f-b215-48d0-990c-adb0538f5fce None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:50.512202 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19e9274f-b215-48d0-990c-adb0538f5fce None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:40:50.513897 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19e9274f-b215-48d0-990c-adb0538f5fce None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:40:50.531614 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 21/41] 10.222.0.52 () {60 vars in 1131 bytes} [Sun Sep 25 00:40:50 2022] POST /identity/v3/roles => generated 225 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:40:50.554844 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9542bb4-9bee-42ea-b645-59202c4c5e6e None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:40:50.579661 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e9274f-b215-48d0-990c-adb0538f5fce None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:40:50.599490 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce72f42-f507-401d-88ba-fee38651d6a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:40:50.600641 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce72f42-f507-401d-88ba-fee38651d6a3 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:50.600927 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce72f42-f507-401d-88ba-fee38651d6a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:50.601508 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce72f42-f507-401d-88ba-fee38651d6a3 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:50.603099 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ce72f42-f507-401d-88ba-fee38651d6a3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:40:50.605429 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ce72f42-f507-401d-88ba-fee38651d6a3 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:40:50.607326 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 21/42] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:40:50 2022] GET /identity/v3/domains/default => generated 199 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:40:50.613173 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce72f42-f507-401d-88ba-fee38651d6a3 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:40:50.647972 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:40:50.723026 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e9274f-b215-48d0-990c-adb0538f5fce None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:40:50.731136 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce72f42-f507-401d-88ba-fee38651d6a3 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:40:50.764120 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e473446e-6787-4df4-947e-cf0890ab119f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:40:50.764752 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e473446e-6787-4df4-947e-cf0890ab119f None admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:50.765028 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e473446e-6787-4df4-947e-cf0890ab119f None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:50.765339 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e473446e-6787-4df4-947e-cf0890ab119f None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:50.766047 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e473446e-6787-4df4-947e-cf0890ab119f None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:40:50.767475 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e473446e-6787-4df4-947e-cf0890ab119f None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:40:50.785489 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 22/43] 10.222.0.52 () {60 vars in 1131 bytes} [Sun Sep 25 00:40:50 2022] POST /identity/v3/roles => generated 223 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 00:40:50.786618 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e473446e-6787-4df4-947e-cf0890ab119f None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:40:50.881002 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3adaf41d-a5a0-4127-9ac5-279762141271 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:40:50.881653 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3adaf41d-a5a0-4127-9ac5-279762141271 None admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:50.881934 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3adaf41d-a5a0-4127-9ac5-279762141271 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:50.882212 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3adaf41d-a5a0-4127-9ac5-279762141271 None admin] PATH_INFO: `/v3/domains/default/users/d6d73e23626043529ac5b9e3889933d4/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:50.884249 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3adaf41d-a5a0-4127-9ac5-279762141271 None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=d6d73e23626043529ac5b9e3889933d4, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:40:50.886695 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3adaf41d-a5a0-4127-9ac5-279762141271 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:40:50.891421 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:40:50.907387 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 22/44] 10.222.0.52 () {58 vars in 1361 bytes} [Sun Sep 25 00:40:50 2022] PUT /identity/v3/domains/default/users/d6d73e23626043529ac5b9e3889933d4/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 295 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:40:50.912048 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e473446e-6787-4df4-947e-cf0890ab119f None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:40:50.921226 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:40:50.960247 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce3809d-1b38-4eb7-8817-dbb704285722 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:40:50.960923 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce3809d-1b38-4eb7-8817-dbb704285722 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:50.961204 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce3809d-1b38-4eb7-8817-dbb704285722 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:50.961505 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce3809d-1b38-4eb7-8817-dbb704285722 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:50.962857 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ce3809d-1b38-4eb7-8817-dbb704285722 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:40:50.965136 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ce3809d-1b38-4eb7-8817-dbb704285722 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:40:50.966962 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 23/45] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:40:50 2022] GET /identity/v3/domains/default => generated 199 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:40:50.973790 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3adaf41d-a5a0-4127-9ac5-279762141271 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:40:51.098960 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3adaf41d-a5a0-4127-9ac5-279762141271 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:40:51.134490 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b1717ba-e64f-4f87-bded-093f35ba036f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:40:51.135009 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1717ba-e64f-4f87-bded-093f35ba036f None admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:51.135285 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1717ba-e64f-4f87-bded-093f35ba036f None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:51.135669 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1717ba-e64f-4f87-bded-093f35ba036f None admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:51.136554 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b1717ba-e64f-4f87-bded-093f35ba036f None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:40:51.138076 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b1717ba-e64f-4f87-bded-093f35ba036f None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:40:51.168157 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 23/46] 10.222.0.52 () {60 vars in 1141 bytes} [Sun Sep 25 00:40:50 2022] POST /identity/v3/projects => generated 312 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:40:53.110142 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685bcf88-dd55-4b7e-b59d-2e28cfc80c5b None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:53.110702 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685bcf88-dd55-4b7e-b59d-2e28cfc80c5b None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:53.110702 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685bcf88-dd55-4b7e-b59d-2e28cfc80c5b None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:53.111991 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 24/47] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:40:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:40:53.117872 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd310f9-93be-4eaf-a0c8-c129d2cc6e30 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:53.118337 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd310f9-93be-4eaf-a0c8-c129d2cc6e30 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:53.118442 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd310f9-93be-4eaf-a0c8-c129d2cc6e30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:53.137648 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-1cd310f9-93be-4eaf-a0c8-c129d2cc6e30 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:40:53.138858 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-1cd310f9-93be-4eaf-a0c8-c129d2cc6e30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:40:53.184491 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:40:53.210141 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 24/48] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:40:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:40:53.345062 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd072c3-2d7b-41f1-99d3-ad401b592ad8 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:53.345387 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd072c3-2d7b-41f1-99d3-ad401b592ad8 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:53.345652 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd072c3-2d7b-41f1-99d3-ad401b592ad8 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:53.347075 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 25/49] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:40:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:40:53.352493 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b88ac6d-96a7-4ff1-974b-5166d6021f32 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:53.352779 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b88ac6d-96a7-4ff1-974b-5166d6021f32 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:53.353053 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b88ac6d-96a7-4ff1-974b-5166d6021f32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:53.372902 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-5b88ac6d-96a7-4ff1-974b-5166d6021f32 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:40:53.374048 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-5b88ac6d-96a7-4ff1-974b-5166d6021f32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:40:53.443093 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 25/50] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:40:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 25 00:40:53.508315 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd072c3-2d7b-41f1-99d3-ad401b592ad8 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:40:53.628562 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd072c3-2d7b-41f1-99d3-ad401b592ad8 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:40:53.665118 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133221d1-7415-4e4a-b1bc-c4c42a1a4411 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:40:53.665797 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133221d1-7415-4e4a-b1bc-c4c42a1a4411 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:53.666077 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133221d1-7415-4e4a-b1bc-c4c42a1a4411 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:53.666349 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133221d1-7415-4e4a-b1bc-c4c42a1a4411 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:53.676747 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-133221d1-7415-4e4a-b1bc-c4c42a1a4411 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:40:53.682233 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-133221d1-7415-4e4a-b1bc-c4c42a1a4411 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:40:53.682233 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:40:53.682233 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:40:53.682233 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:40:53.682233 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:40:53.682233 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:40:53.682233 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:40:53.682233 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:40:53.682233 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:40:53.682233 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:40:53.682233 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:40:53.682233 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:40:53.682233 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:40:53.682233 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:40:53.682233 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:40:53.682233 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:40:53.683507 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:40:53.683507 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:40:53.683507 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:40:53.683507 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:40:53.683507 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:40:53.683507 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:40:53.683507 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:40:53.683507 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:40:53.683507 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:40:53.683507 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:40:53.683507 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:40:53.683507 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:40:53.683507 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:40:53.683507 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:40:53.683507 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:40:53.683507 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:40:53.684432 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:40:53.684432 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:40:53.684432 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:40:53.684432 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:40:53.684432 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:40:53.684432 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:40:53.684432 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:40:53.684432 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:40:53.684432 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:40:53.684432 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:40:53.684432 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:40:53.684432 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:40:53.684432 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 26/51] 10.222.0.52 () {56 vars in 1096 bytes} [Sun Sep 25 00:40:53 2022] GET /identity/v3/roles/admin => generated 83 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:40:53.691251 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b88ac6d-96a7-4ff1-974b-5166d6021f32 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:40:53.756593 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133221d1-7415-4e4a-b1bc-c4c42a1a4411 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:40:53.792125 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b88ac6d-96a7-4ff1-974b-5166d6021f32 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:40:53.792497 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:40:53.802225 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:40:53.829497 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a34b52-9f5d-433c-b2eb-2d5dbc73da2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:40:53.830115 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a34b52-9f5d-433c-b2eb-2d5dbc73da2d None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:53.830394 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a34b52-9f5d-433c-b2eb-2d5dbc73da2d None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:53.830667 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a34b52-9f5d-433c-b2eb-2d5dbc73da2d None admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:53.831365 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a34b52-9f5d-433c-b2eb-2d5dbc73da2d None admin] RBAC: Adding query filter params (name=admin) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:40:53.831696 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a34b52-9f5d-433c-b2eb-2d5dbc73da2d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:40:53.833085 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a34b52-9f5d-433c-b2eb-2d5dbc73da2d None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:40:53.845751 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 26/52] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:40:53 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:40:53.851404 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a34b52-9f5d-433c-b2eb-2d5dbc73da2d None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:40:53.865632 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133221d1-7415-4e4a-b1bc-c4c42a1a4411 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:40:53.902457 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce0dc0e-b7b2-4782-b954-ebeaefceda4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:40:53.903000 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce0dc0e-b7b2-4782-b954-ebeaefceda4b None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:53.903283 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce0dc0e-b7b2-4782-b954-ebeaefceda4b None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:53.903551 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce0dc0e-b7b2-4782-b954-ebeaefceda4b None admin] PATH_INFO: `/v3/domains/default` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:53.904689 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ce0dc0e-b7b2-4782-b954-ebeaefceda4b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:40:53.906182 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ce0dc0e-b7b2-4782-b954-ebeaefceda4b None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:40:53.907739 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 27/53] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:40:53 2022] GET /identity/v3/domains/default => generated 199 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:40:53.913273 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce0dc0e-b7b2-4782-b954-ebeaefceda4b None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:40:53.956861 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a34b52-9f5d-433c-b2eb-2d5dbc73da2d None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:40:53.994881 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a4e445f-5f3d-420c-9320-3a5418cc2429 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:40:53.995512 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4e445f-5f3d-420c-9320-3a5418cc2429 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:53.995787 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4e445f-5f3d-420c-9320-3a5418cc2429 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:53.996057 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4e445f-5f3d-420c-9320-3a5418cc2429 None admin] PATH_INFO: `/v3/users/d6d73e23626043529ac5b9e3889933d4` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:53.997869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a4e445f-5f3d-420c-9320-3a5418cc2429 None admin] RBAC: Authorizing `identity:get_user(user_id=d6d73e23626043529ac5b9e3889933d4)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:40:53.999313 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a4e445f-5f3d-420c-9320-3a5418cc2429 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:40:54.000896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 27/54] 10.222.0.52 () {56 vars in 1177 bytes} [Sun Sep 25 00:40:53 2022] GET /identity/v3/users/d6d73e23626043529ac5b9e3889933d4 => generated 247 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:40:54.006812 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a4e445f-5f3d-420c-9320-3a5418cc2429 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:40:54.013333 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce0dc0e-b7b2-4782-b954-ebeaefceda4b None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:40:54.014869 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:40:54.047164 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:40:54.048549 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae0d706-d2e1-4fd1-80df-f8c92bafead9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:40:54.049152 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae0d706-d2e1-4fd1-80df-f8c92bafead9 None admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:54.049466 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae0d706-d2e1-4fd1-80df-f8c92bafead9 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:54.049744 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae0d706-d2e1-4fd1-80df-f8c92bafead9 None admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:54.050391 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ae0d706-d2e1-4fd1-80df-f8c92bafead9 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:40:54.051828 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ae0d706-d2e1-4fd1-80df-f8c92bafead9 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:40:54.079036 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 28/55] 10.222.0.52 () {60 vars in 1140 bytes} [Sun Sep 25 00:40:53 2022] POST /identity/v3/projects => generated 298 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 00:40:54.117949 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a4e445f-5f3d-420c-9320-3a5418cc2429 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:40:54.156375 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f47779-2439-4cd4-9f93-76349a679f20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:40:54.156935 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f47779-2439-4cd4-9f93-76349a679f20 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:54.157207 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f47779-2439-4cd4-9f93-76349a679f20 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:54.157499 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f47779-2439-4cd4-9f93-76349a679f20 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:54.158996 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f47779-2439-4cd4-9f93-76349a679f20 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:40:54.160456 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f47779-2439-4cd4-9f93-76349a679f20 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:40:54.162128 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 28/56] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:40:54 2022] GET /identity/v3/domains/default => generated 199 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:40:54.169505 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae0d706-d2e1-4fd1-80df-f8c92bafead9 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:40:54.278220 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae0d706-d2e1-4fd1-80df-f8c92bafead9 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:40:54.313984 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e166d68-286f-4111-84bf-89e05f5a2b6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:40:54.314597 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e166d68-286f-4111-84bf-89e05f5a2b6a None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:54.314874 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e166d68-286f-4111-84bf-89e05f5a2b6a None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:54.315144 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e166d68-286f-4111-84bf-89e05f5a2b6a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:54.315870 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e166d68-286f-4111-84bf-89e05f5a2b6a None admin] RBAC: Adding query filter params (role.id=2c242abf6ed8410ab0a4a354ca1794fb, scope.domain.id=default, user.id=d6d73e23626043529ac5b9e3889933d4) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:40:54.316179 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e166d68-286f-4111-84bf-89e05f5a2b6a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:40:54.317660 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e166d68-286f-4111-84bf-89e05f5a2b6a None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:40:54.326628 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 29/57] 10.222.0.52 () {56 vars in 1322 bytes} [Sun Sep 25 00:40:54 2022] GET /identity/v3/role_assignments?user.id=d6d73e23626043529ac5b9e3889933d4&scope.domain.id=default&role.id=2c242abf6ed8410ab0a4a354ca1794fb => generated 530 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:40:56.299116 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc761366-68d0-48cb-9b30-8a9cd7f0509c None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:56.299639 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc761366-68d0-48cb-9b30-8a9cd7f0509c None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:56.299639 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc761366-68d0-48cb-9b30-8a9cd7f0509c None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:56.300982 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 29/58] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:40:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:40:56.306297 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb672497-dc98-4499-af84-90452acfe93d None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:56.306704 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb672497-dc98-4499-af84-90452acfe93d None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:56.306849 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb672497-dc98-4499-af84-90452acfe93d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:56.428368 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-eb672497-dc98-4499-af84-90452acfe93d None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:40:56.429628 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-eb672497-dc98-4499-af84-90452acfe93d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:40:56.443674 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:40:56.498016 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 30/59] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:40:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:40:56.764246 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc761366-68d0-48cb-9b30-8a9cd7f0509c None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:40:56.864724 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc761366-68d0-48cb-9b30-8a9cd7f0509c None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:40:56.872532 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:40:56.899287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bfb0742-2da8-4fc7-bcfb-b8db6fa2acb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:40:56.899908 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfb0742-2da8-4fc7-bcfb-b8db6fa2acb9 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:56.900185 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfb0742-2da8-4fc7-bcfb-b8db6fa2acb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:56.900452 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfb0742-2da8-4fc7-bcfb-b8db6fa2acb9 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:56.901607 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bfb0742-2da8-4fc7-bcfb-b8db6fa2acb9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:40:56.903037 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bfb0742-2da8-4fc7-bcfb-b8db6fa2acb9 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:40:56.904612 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 30/60] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:40:56 2022] GET /identity/v3/domains/default => generated 199 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:40:56.910125 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb672497-dc98-4499-af84-90452acfe93d None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:40:57.008251 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb672497-dc98-4499-af84-90452acfe93d None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:40:57.042608 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1adc4e44-1fba-4c82-8419-172a76520eb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:40:57.043119 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1adc4e44-1fba-4c82-8419-172a76520eb6 None admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:40:57.043395 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1adc4e44-1fba-4c82-8419-172a76520eb6 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:40:57.043664 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1adc4e44-1fba-4c82-8419-172a76520eb6 None admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:40:57.044306 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1adc4e44-1fba-4c82-8419-172a76520eb6 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:40:57.045769 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1adc4e44-1fba-4c82-8419-172a76520eb6 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:40:57.071189 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 31/61] 10.222.0.52 () {60 vars in 1132 bytes} [Sun Sep 25 00:40:56 2022] POST /identity/v3/users => generated 275 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:41:00.208354 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c90cf7-69d3-427a-a89e-a016f2c81d9c None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:00.209413 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c90cf7-69d3-427a-a89e-a016f2c81d9c None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:00.209920 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c90cf7-69d3-427a-a89e-a016f2c81d9c None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:00.211459 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 31/62] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:00.216963 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccef0cd0-be29-49fe-9263-50b21550e5a8 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:00.217809 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccef0cd0-be29-49fe-9263-50b21550e5a8 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:00.218278 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccef0cd0-be29-49fe-9263-50b21550e5a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:00.229460 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:00.242902 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-ccef0cd0-be29-49fe-9263-50b21550e5a8 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:00.244158 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-ccef0cd0-be29-49fe-9263-50b21550e5a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:00.338832 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 32/63] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:41:00.413058 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305735ce-5376-4c2d-b252-6ece5e986d64 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:00.413368 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305735ce-5376-4c2d-b252-6ece5e986d64 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:00.413836 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305735ce-5376-4c2d-b252-6ece5e986d64 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:00.415331 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 32/64] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:00.421215 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b34204c-1ab6-4c21-a2f2-2aa95b1760c5 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:00.421691 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b34204c-1ab6-4c21-a2f2-2aa95b1760c5 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:00.422125 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b34204c-1ab6-4c21-a2f2-2aa95b1760c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:00.449041 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-8b34204c-1ab6-4c21-a2f2-2aa95b1760c5 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:00.450556 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-8b34204c-1ab6-4c21-a2f2-2aa95b1760c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:00.545688 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 33/65] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:41:00.801275 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-305735ce-5376-4c2d-b252-6ece5e986d64 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:00.882882 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b34204c-1ab6-4c21-a2f2-2aa95b1760c5 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:00.939037 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-305735ce-5376-4c2d-b252-6ece5e986d64 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:01.022876 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faacd15c-dcb4-47e2-a34f-ee14fb41547e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:01.024969 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faacd15c-dcb4-47e2-a34f-ee14fb41547e None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:01.025545 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faacd15c-dcb4-47e2-a34f-ee14fb41547e None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:01.026648 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:01.029495 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faacd15c-dcb4-47e2-a34f-ee14fb41547e None admin] PATH_INFO: `/v3/roles/member` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:01.058859 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faacd15c-dcb4-47e2-a34f-ee14fb41547e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:41:01.063991 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-faacd15c-dcb4-47e2-a34f-ee14fb41547e None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 25 00:41:01.063991 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:41:01.063991 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:41:01.063991 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:41:01.063991 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:41:01.063991 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:41:01.063991 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:41:01.063991 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:41:01.063991 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:41:01.063991 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:41:01.063991 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:41:01.063991 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:41:01.063991 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:41:01.063991 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:41:01.063991 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:41:01.063991 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:41:01.064867 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:41:01.064867 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:41:01.064867 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:41:01.064867 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:41:01.064867 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:41:01.064867 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:41:01.064867 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:41:01.064867 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:41:01.064867 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:41:01.064867 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:41:01.064867 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:41:01.064867 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:41:01.064867 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:41:01.064867 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:41:01.064867 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:41:01.064867 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:41:01.065676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:41:01.065676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:41:01.065676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:41:01.065676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:41:01.065676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:41:01.065676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:41:01.065676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:41:01.065676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:41:01.065676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:41:01.065676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:41:01.065676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 25 00:41:01.065676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:41:01.068285 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 33/66] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:41:00 2022] GET /identity/v3/roles/member => generated 84 bytes in 267 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:41:01.070434 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d871ead3-a1be-4f76-95fa-82e37f3fa79c None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:01.070936 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d871ead3-a1be-4f76-95fa-82e37f3fa79c None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:01.071421 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d871ead3-a1be-4f76-95fa-82e37f3fa79c None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:01.074176 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 34/67] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:01 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:01.075230 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d871ead3-a1be-4f76-95fa-82e37f3fa79c None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:01.086585 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b34204c-1ab6-4c21-a2f2-2aa95b1760c5 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:01.116686 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:01.131358 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab1e0fa-15ce-4dec-8c35-3626192bfafe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:01.132353 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab1e0fa-15ce-4dec-8c35-3626192bfafe None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:01.132777 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab1e0fa-15ce-4dec-8c35-3626192bfafe None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:01.133303 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab1e0fa-15ce-4dec-8c35-3626192bfafe None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:01.146084 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cab1e0fa-15ce-4dec-8c35-3626192bfafe None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:41:01.151605 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-cab1e0fa-15ce-4dec-8c35-3626192bfafe None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:41:01.151605 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:41:01.151605 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:41:01.151605 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:41:01.151605 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:41:01.151605 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:41:01.151605 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:41:01.151605 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:41:01.151605 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:41:01.151605 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:41:01.151605 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:41:01.151605 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:41:01.151605 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:41:01.151605 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:41:01.151605 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:41:01.151605 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:41:01.152482 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:41:01.152482 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:41:01.152482 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:41:01.152482 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:41:01.152482 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:41:01.152482 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:41:01.152482 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:41:01.152482 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:41:01.152482 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:41:01.152482 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:41:01.152482 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:41:01.152482 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:41:01.152482 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:41:01.152482 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:41:01.152482 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:41:01.152482 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:41:01.153697 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:41:01.153697 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:41:01.153697 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:41:01.153697 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:41:01.153697 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:41:01.153697 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:41:01.153697 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:41:01.153697 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:41:01.153697 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:41:01.153697 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:41:01.153697 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:41:01.153697 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:41:01.153697 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 34/68] 10.222.0.52 () {56 vars in 1096 bytes} [Sun Sep 25 00:41:00 2022] GET /identity/v3/roles/admin => generated 83 bytes in 270 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:41:01.160077 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e29c24-f0cd-492d-a1a3-27ddabfe0af9 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:01.160366 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e29c24-f0cd-492d-a1a3-27ddabfe0af9 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:01.160641 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e29c24-f0cd-492d-a1a3-27ddabfe0af9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:01.178653 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-68e29c24-f0cd-492d-a1a3-27ddabfe0af9 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:01.179865 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-68e29c24-f0cd-492d-a1a3-27ddabfe0af9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:01.193518 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d871ead3-a1be-4f76-95fa-82e37f3fa79c None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:01.240853 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e281f0af-357c-4a75-a8c0-2b2907db4b43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:01.241575 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e281f0af-357c-4a75-a8c0-2b2907db4b43 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:01.241759 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e281f0af-357c-4a75-a8c0-2b2907db4b43 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:01.242027 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e281f0af-357c-4a75-a8c0-2b2907db4b43 None admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:01.242688 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e281f0af-357c-4a75-a8c0-2b2907db4b43 None admin] RBAC: Adding query filter params (name=member) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:01.242994 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e281f0af-357c-4a75-a8c0-2b2907db4b43 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:01.244416 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e281f0af-357c-4a75-a8c0-2b2907db4b43 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:01.250805 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 35/69] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:41:01.252054 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c78c0d-f010-4cf0-a7e2-7ae964ece7b7 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:01.252335 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c78c0d-f010-4cf0-a7e2-7ae964ece7b7 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:01.252605 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c78c0d-f010-4cf0-a7e2-7ae964ece7b7 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:01.253873 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 36/70] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:01.257467 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c78c0d-f010-4cf0-a7e2-7ae964ece7b7 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:01.261583 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 35/71] 10.222.0.52 () {56 vars in 1101 bytes} [Sun Sep 25 00:41:01 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:01.263175 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-751c6d45-cea4-426b-8f80-dba8db5f8af0 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:01.263175 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-751c6d45-cea4-426b-8f80-dba8db5f8af0 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:01.263425 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-751c6d45-cea4-426b-8f80-dba8db5f8af0 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:01.268583 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 36/72] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:01.268583 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f840d4-3268-4255-a6b4-3ea5f22148c6 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:01.268583 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f840d4-3268-4255-a6b4-3ea5f22148c6 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:01.268583 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f840d4-3268-4255-a6b4-3ea5f22148c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:01.284911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-e3f840d4-3268-4255-a6b4-3ea5f22148c6 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:01.286141 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-e3f840d4-3268-4255-a6b4-3ea5f22148c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:01.326040 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:01.358019 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 37/73] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:41:01.370412 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f840d4-3268-4255-a6b4-3ea5f22148c6 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:01.380565 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c78c0d-f010-4cf0-a7e2-7ae964ece7b7 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:01.399903 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:01.421842 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e40f373d-1020-40e8-8c9d-b5c18b6b76f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:01.422786 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e40f373d-1020-40e8-8c9d-b5c18b6b76f8 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:01.423363 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e40f373d-1020-40e8-8c9d-b5c18b6b76f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:01.423836 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e40f373d-1020-40e8-8c9d-b5c18b6b76f8 None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:01.424770 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e40f373d-1020-40e8-8c9d-b5c18b6b76f8 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:01.425316 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e40f373d-1020-40e8-8c9d-b5c18b6b76f8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:01.427063 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e40f373d-1020-40e8-8c9d-b5c18b6b76f8 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:01.437350 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 37/74] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:41:01 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:01.443565 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d01ea6-ac20-48d9-92a2-a990fa6d58be None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:01.444111 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d01ea6-ac20-48d9-92a2-a990fa6d58be None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:01.444706 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d01ea6-ac20-48d9-92a2-a990fa6d58be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:01.465320 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-b7d01ea6-ac20-48d9-92a2-a990fa6d58be None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:01.466873 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-b7d01ea6-ac20-48d9-92a2-a990fa6d58be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:01.476884 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f840d4-3268-4255-a6b4-3ea5f22148c6 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:01.529459 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c42eeb-8a3f-49fa-bc7c-a213d0ea2e74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:01.531420 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c42eeb-8a3f-49fa-bc7c-a213d0ea2e74 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:01.533215 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c42eeb-8a3f-49fa-bc7c-a213d0ea2e74 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:01.533738 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c42eeb-8a3f-49fa-bc7c-a213d0ea2e74 None admin] PATH_INFO: `/v3/users/6980cb57558a43d2b384c7b561d7af70` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:01.548051 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 38/75] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:41:01.549202 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d01ea6-ac20-48d9-92a2-a990fa6d58be None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:01.553719 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24c42eeb-8a3f-49fa-bc7c-a213d0ea2e74 None admin] RBAC: Authorizing `identity:get_user(user_id=6980cb57558a43d2b384c7b561d7af70)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:01.555203 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24c42eeb-8a3f-49fa-bc7c-a213d0ea2e74 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:01.557747 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 38/76] 10.222.0.52 () {56 vars in 1177 bytes} [Sun Sep 25 00:41:01 2022] GET /identity/v3/users/6980cb57558a43d2b384c7b561d7af70 => generated 275 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:01.558656 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c42eeb-8a3f-49fa-bc7c-a213d0ea2e74 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:01.610441 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:01.672539 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d01ea6-ac20-48d9-92a2-a990fa6d58be None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:01.680375 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:01.701154 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c42eeb-8a3f-49fa-bc7c-a213d0ea2e74 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:01.713396 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746de5a6-a9da-4931-ad21-7c0981dc65dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:01.714191 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746de5a6-a9da-4931-ad21-7c0981dc65dc None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:01.714474 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746de5a6-a9da-4931-ad21-7c0981dc65dc None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:01.714747 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746de5a6-a9da-4931-ad21-7c0981dc65dc None admin] PATH_INFO: `/v3/users/d6d73e23626043529ac5b9e3889933d4` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:01.715957 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-746de5a6-a9da-4931-ad21-7c0981dc65dc None admin] RBAC: Authorizing `identity:get_user(user_id=d6d73e23626043529ac5b9e3889933d4)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:01.717434 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-746de5a6-a9da-4931-ad21-7c0981dc65dc None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:01.723843 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 39/77] 10.222.0.52 () {56 vars in 1177 bytes} [Sun Sep 25 00:41:01 2022] GET /identity/v3/users/d6d73e23626043529ac5b9e3889933d4 => generated 247 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:01.723843 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746de5a6-a9da-4931-ad21-7c0981dc65dc None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:01.744028 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6bd78bb-3c8e-4d32-9316-63e6a5b0bdbc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:01.745361 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6bd78bb-3c8e-4d32-9316-63e6a5b0bdbc None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:01.745644 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6bd78bb-3c8e-4d32-9316-63e6a5b0bdbc None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:01.746251 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6bd78bb-3c8e-4d32-9316-63e6a5b0bdbc None admin] PATH_INFO: `/v3/domains/default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:01.747410 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6bd78bb-3c8e-4d32-9316-63e6a5b0bdbc None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:01.749057 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6bd78bb-3c8e-4d32-9316-63e6a5b0bdbc None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:01.750445 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 39/78] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:41:01 2022] GET /identity/v3/domains/default => generated 199 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:01.751292 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6bd78bb-3c8e-4d32-9316-63e6a5b0bdbc None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:01.826385 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746de5a6-a9da-4931-ad21-7c0981dc65dc None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:01.841455 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:01.855275 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6bd78bb-3c8e-4d32-9316-63e6a5b0bdbc None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:01.862233 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b447f11-4fcb-4154-a88c-179ccd6c575a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:01.862776 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b447f11-4fcb-4154-a88c-179ccd6c575a None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:01.863319 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b447f11-4fcb-4154-a88c-179ccd6c575a None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:01.863585 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b447f11-4fcb-4154-a88c-179ccd6c575a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:01.891036 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba82a0ff-d2a9-4c53-b1e6-13fdbbeeea09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:01.891664 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba82a0ff-d2a9-4c53-b1e6-13fdbbeeea09 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:01.891948 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba82a0ff-d2a9-4c53-b1e6-13fdbbeeea09 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:01.892219 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba82a0ff-d2a9-4c53-b1e6-13fdbbeeea09 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:01.902572 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba82a0ff-d2a9-4c53-b1e6-13fdbbeeea09 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:41:01.907497 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-ba82a0ff-d2a9-4c53-b1e6-13fdbbeeea09 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 25 00:41:01.907497 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:41:01.907497 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:41:01.907497 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:41:01.907497 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:41:01.907497 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:41:01.907497 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:41:01.907497 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:41:01.907497 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:41:01.907497 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:41:01.907497 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:41:01.907497 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:41:01.907497 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:41:01.907497 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:41:01.907497 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:41:01.907497 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:41:01.908415 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:41:01.908415 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:41:01.908415 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:41:01.908415 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:41:01.908415 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:41:01.908415 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:41:01.908415 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:41:01.908415 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:41:01.908415 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:41:01.908415 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:41:01.908415 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:41:01.908415 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:41:01.908415 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:41:01.908415 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:41:01.908415 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:41:01.908415 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:41:01.909165 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:41:01.909165 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:41:01.909165 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:41:01.909165 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:41:01.909165 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:41:01.909165 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:41:01.909165 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:41:01.909165 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:41:01.909165 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:41:01.909165 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:41:01.909165 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 25 00:41:01.909165 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:41:01.909165 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 40/79] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:41:01 2022] GET /identity/v3/roles/member => generated 84 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:41:01.910577 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba82a0ff-d2a9-4c53-b1e6-13fdbbeeea09 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:01.926046 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:01.935695 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b447f11-4fcb-4154-a88c-179ccd6c575a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:01.937186 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b447f11-4fcb-4154-a88c-179ccd6c575a None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:02.012601 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba82a0ff-d2a9-4c53-b1e6-13fdbbeeea09 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:02.037624 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 40/80] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:41:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 00:41:02.038445 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b447f11-4fcb-4154-a88c-179ccd6c575a None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:02.051418 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb326e2f-88d2-4df6-b98a-9fda499855f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:02.052019 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb326e2f-88d2-4df6-b98a-9fda499855f1 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:02.052299 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb326e2f-88d2-4df6-b98a-9fda499855f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:02.052568 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb326e2f-88d2-4df6-b98a-9fda499855f1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:02.053239 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:02.121556 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb326e2f-88d2-4df6-b98a-9fda499855f1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:02.122983 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb326e2f-88d2-4df6-b98a-9fda499855f1 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:02.137703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:02.140642 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b447f11-4fcb-4154-a88c-179ccd6c575a None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:02.177085 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d350da9e-2af6-4947-9231-a22490d3eb7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:02.177722 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d350da9e-2af6-4947-9231-a22490d3eb7d None admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:02.177996 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d350da9e-2af6-4947-9231-a22490d3eb7d None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:02.178266 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d350da9e-2af6-4947-9231-a22490d3eb7d None admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:02.178914 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d350da9e-2af6-4947-9231-a22490d3eb7d None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:02.180339 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d350da9e-2af6-4947-9231-a22490d3eb7d None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:02.202346 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 41/81] 10.222.0.52 () {60 vars in 1132 bytes} [Sun Sep 25 00:41:02 2022] POST /identity/v3/users => generated 289 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:41:02.212552 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d350da9e-2af6-4947-9231-a22490d3eb7d None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:02.233837 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 41/82] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:41:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 00:41:02.234901 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb326e2f-88d2-4df6-b98a-9fda499855f1 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:02.296119 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:02.334728 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d350da9e-2af6-4947-9231-a22490d3eb7d None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:02.356989 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb326e2f-88d2-4df6-b98a-9fda499855f1 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:02.373483 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:02.390126 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb6c922b-2754-4d83-8498-fa2c8294b54f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:02.390762 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6c922b-2754-4d83-8498-fa2c8294b54f None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:02.391037 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6c922b-2754-4d83-8498-fa2c8294b54f None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:02.391302 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6c922b-2754-4d83-8498-fa2c8294b54f None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:02.403780 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b49d4505-4ee3-4688-bfb3-4cf91c5ebd38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:02.404396 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49d4505-4ee3-4688-bfb3-4cf91c5ebd38 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:02.404677 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49d4505-4ee3-4688-bfb3-4cf91c5ebd38 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:02.404951 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49d4505-4ee3-4688-bfb3-4cf91c5ebd38 None admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:02.405659 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b49d4505-4ee3-4688-bfb3-4cf91c5ebd38 None admin] RBAC: Adding query filter params (name=member) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:02.405961 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b49d4505-4ee3-4688-bfb3-4cf91c5ebd38 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:02.406216 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb6c922b-2754-4d83-8498-fa2c8294b54f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:41:02.407358 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b49d4505-4ee3-4688-bfb3-4cf91c5ebd38 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:02.412192 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-fb6c922b-2754-4d83-8498-fa2c8294b54f None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 00:41:02.412192 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:41:02.412192 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:41:02.412192 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:41:02.412192 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:41:02.412192 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:41:02.412192 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:41:02.412192 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:41:02.412192 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:41:02.412192 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:41:02.412192 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:41:02.412192 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:41:02.412192 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:41:02.412192 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:41:02.412192 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:41:02.412192 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:41:02.413038 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:41:02.413038 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:41:02.413038 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:41:02.413038 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:41:02.413038 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:41:02.413038 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:41:02.413038 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:41:02.413038 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:41:02.413038 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:41:02.413038 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:41:02.413038 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:41:02.413038 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:41:02.413038 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:41:02.413038 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:41:02.413038 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:41:02.413038 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:41:02.414750 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:41:02.414750 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:41:02.414750 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:41:02.414750 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:41:02.414750 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:41:02.414750 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:41:02.414750 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:41:02.414750 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:41:02.414750 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:41:02.414750 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:41:02.414750 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 00:41:02.414750 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:41:02.414750 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 42/83] 10.222.0.52 () {56 vars in 1114 bytes} [Sun Sep 25 00:41:02 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:41:02.414750 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb6c922b-2754-4d83-8498-fa2c8294b54f None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:02.421295 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 42/84] 10.222.0.52 () {56 vars in 1101 bytes} [Sun Sep 25 00:41:02 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:02.422108 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b49d4505-4ee3-4688-bfb3-4cf91c5ebd38 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:02.550333 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb6c922b-2754-4d83-8498-fa2c8294b54f None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:02.561725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b49d4505-4ee3-4688-bfb3-4cf91c5ebd38 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:02.587281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:02.595507 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb356764-583f-44af-9a55-16cc0991fed2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:02.596344 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb356764-583f-44af-9a55-16cc0991fed2 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:02.596760 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb356764-583f-44af-9a55-16cc0991fed2 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:02.597199 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb356764-583f-44af-9a55-16cc0991fed2 None admin] PATH_INFO: `/v3/projects/cd69c40698884b3192bb19ba6968842d` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:02.598438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb356764-583f-44af-9a55-16cc0991fed2 None admin] RBAC: Authorizing `identity:get_project(project_id=cd69c40698884b3192bb19ba6968842d)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:02.600010 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb356764-583f-44af-9a55-16cc0991fed2 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:02.605400 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 43/85] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:41:02 2022] GET /identity/v3/projects/cd69c40698884b3192bb19ba6968842d => generated 298 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:02.605546 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46765b9e-a824-441f-8906-abbbbcd5c020 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:02.605672 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb356764-583f-44af-9a55-16cc0991fed2 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:02.605757 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46765b9e-a824-441f-8906-abbbbcd5c020 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:02.605757 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46765b9e-a824-441f-8906-abbbbcd5c020 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:02.605757 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46765b9e-a824-441f-8906-abbbbcd5c020 None admin] PATH_INFO: `/v3/projects/cd69c40698884b3192bb19ba6968842d` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:02.606447 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46765b9e-a824-441f-8906-abbbbcd5c020 None admin] RBAC: Authorizing `identity:get_project(project_id=cd69c40698884b3192bb19ba6968842d)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:02.607933 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46765b9e-a824-441f-8906-abbbbcd5c020 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:02.610327 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 43/86] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:41:02 2022] GET /identity/v3/projects/cd69c40698884b3192bb19ba6968842d => generated 298 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:02.610447 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46765b9e-a824-441f-8906-abbbbcd5c020 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:02.670530 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:02.740625 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46765b9e-a824-441f-8906-abbbbcd5c020 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:02.742684 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb356764-583f-44af-9a55-16cc0991fed2 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:02.781633 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50516c54-28cd-4340-b511-ef8bcf45bdbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:02.782766 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50516c54-28cd-4340-b511-ef8bcf45bdbb None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:02.783326 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50516c54-28cd-4340-b511-ef8bcf45bdbb None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:02.783943 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50516c54-28cd-4340-b511-ef8bcf45bdbb None admin] PATH_INFO: `/v3/users/6980cb57558a43d2b384c7b561d7af70` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:02.785053 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea2b88b2-f8c6-455f-9ef3-0dc1c596634f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:02.785634 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50516c54-28cd-4340-b511-ef8bcf45bdbb None admin] RBAC: Authorizing `identity:get_user(user_id=6980cb57558a43d2b384c7b561d7af70)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:02.785736 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea2b88b2-f8c6-455f-9ef3-0dc1c596634f None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:02.785961 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea2b88b2-f8c6-455f-9ef3-0dc1c596634f None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:02.786232 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea2b88b2-f8c6-455f-9ef3-0dc1c596634f None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:02.786896 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea2b88b2-f8c6-455f-9ef3-0dc1c596634f None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:02.787198 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea2b88b2-f8c6-455f-9ef3-0dc1c596634f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:02.788248 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50516c54-28cd-4340-b511-ef8bcf45bdbb None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:02.788569 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea2b88b2-f8c6-455f-9ef3-0dc1c596634f None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:02.790432 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 44/87] 10.222.0.52 () {56 vars in 1177 bytes} [Sun Sep 25 00:41:02 2022] GET /identity/v3/users/6980cb57558a43d2b384c7b561d7af70 => generated 275 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:02.792076 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50516c54-28cd-4340-b511-ef8bcf45bdbb None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:02.799437 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 44/88] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:41:02 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:02.800407 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea2b88b2-f8c6-455f-9ef3-0dc1c596634f None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:02.874382 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:02.907150 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50516c54-28cd-4340-b511-ef8bcf45bdbb None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:02.920087 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:02.922305 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea2b88b2-f8c6-455f-9ef3-0dc1c596634f None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:02.950436 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcfaf180-d714-4855-9711-71fff8f00734 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:02.951052 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcfaf180-d714-4855-9711-71fff8f00734 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:02.951383 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcfaf180-d714-4855-9711-71fff8f00734 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:02.951658 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcfaf180-d714-4855-9711-71fff8f00734 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:02.952518 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcfaf180-d714-4855-9711-71fff8f00734 None admin] RBAC: Adding query filter params (role.id=40f54f5ac0994c51960ac413da8d2cc9, scope.project.id=cd69c40698884b3192bb19ba6968842d, user.id=6980cb57558a43d2b384c7b561d7af70) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:02.952855 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcfaf180-d714-4855-9711-71fff8f00734 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:02.954501 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcfaf180-d714-4855-9711-71fff8f00734 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:02.961019 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 45/89] 10.222.0.52 () {56 vars in 1374 bytes} [Sun Sep 25 00:41:02 2022] GET /identity/v3/role_assignments?user.id=6980cb57558a43d2b384c7b561d7af70&scope.project.id=cd69c40698884b3192bb19ba6968842d&role.id=40f54f5ac0994c51960ac413da8d2cc9 => generated 259 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:02.961187 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a0fca6b-4f0e-4ff4-804c-db0c0fe0bb12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:02.961651 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcfaf180-d714-4855-9711-71fff8f00734 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:02.966084 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0fca6b-4f0e-4ff4-804c-db0c0fe0bb12 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:02.966360 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0fca6b-4f0e-4ff4-804c-db0c0fe0bb12 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:02.966627 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0fca6b-4f0e-4ff4-804c-db0c0fe0bb12 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:02.967375 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a0fca6b-4f0e-4ff4-804c-db0c0fe0bb12 None admin] RBAC: Adding query filter params (role.id=2c242abf6ed8410ab0a4a354ca1794fb, scope.project.id=cd69c40698884b3192bb19ba6968842d, user.id=d6d73e23626043529ac5b9e3889933d4) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:02.967685 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a0fca6b-4f0e-4ff4-804c-db0c0fe0bb12 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:02.969369 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a0fca6b-4f0e-4ff4-804c-db0c0fe0bb12 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:02.986402 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 45/90] 10.222.0.52 () {56 vars in 1374 bytes} [Sun Sep 25 00:41:02 2022] GET /identity/v3/role_assignments?user.id=d6d73e23626043529ac5b9e3889933d4&scope.project.id=cd69c40698884b3192bb19ba6968842d&role.id=2c242abf6ed8410ab0a4a354ca1794fb => generated 259 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:02.987513 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a0fca6b-4f0e-4ff4-804c-db0c0fe0bb12 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:03.207652 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcfaf180-d714-4855-9711-71fff8f00734 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:03.232629 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a0fca6b-4f0e-4ff4-804c-db0c0fe0bb12 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:03.262235 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:03.279662 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff490f6c-f250-41f9-bc1e-f1ef197799cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:03.281308 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff490f6c-f250-41f9-bc1e-f1ef197799cd None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:03.281662 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff490f6c-f250-41f9-bc1e-f1ef197799cd None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:03.281941 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff490f6c-f250-41f9-bc1e-f1ef197799cd None admin] PATH_INFO: `/v3/users/6980cb57558a43d2b384c7b561d7af70` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:03.285754 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff490f6c-f250-41f9-bc1e-f1ef197799cd None admin] RBAC: Authorizing `identity:get_user(user_id=6980cb57558a43d2b384c7b561d7af70)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:03.287307 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff490f6c-f250-41f9-bc1e-f1ef197799cd None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:03.289111 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 46/91] 10.222.0.52 () {56 vars in 1177 bytes} [Sun Sep 25 00:41:02 2022] GET /identity/v3/users/6980cb57558a43d2b384c7b561d7af70 => generated 275 bytes in 302 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:03.297715 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cdffea5-62a8-4f99-afa9-835d8ddf6b43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:03.298349 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cdffea5-62a8-4f99-afa9-835d8ddf6b43 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:03.298630 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cdffea5-62a8-4f99-afa9-835d8ddf6b43 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:03.298903 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cdffea5-62a8-4f99-afa9-835d8ddf6b43 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:03.318442 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff490f6c-f250-41f9-bc1e-f1ef197799cd None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:03.401176 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:03.441315 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cdffea5-62a8-4f99-afa9-835d8ddf6b43 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:03.442809 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cdffea5-62a8-4f99-afa9-835d8ddf6b43 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:03.516728 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff490f6c-f250-41f9-bc1e-f1ef197799cd None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:03.604534 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 46/92] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:41:02 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 643 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 00:41:03.610200 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cdffea5-62a8-4f99-afa9-835d8ddf6b43 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:03.631340 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db2b226-6bcb-4175-a394-ec836b8fccd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:03.632340 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db2b226-6bcb-4175-a394-ec836b8fccd4 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:03.632788 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db2b226-6bcb-4175-a394-ec836b8fccd4 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:03.633268 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db2b226-6bcb-4175-a394-ec836b8fccd4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:03.746694 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:03.792568 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cdffea5-62a8-4f99-afa9-835d8ddf6b43 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:03.846627 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11fc342a-4a0e-4ea9-ad4d-e6961af25823 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:03.847629 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fc342a-4a0e-4ea9-ad4d-e6961af25823 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:03.848101 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fc342a-4a0e-4ea9-ad4d-e6961af25823 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:03.848590 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fc342a-4a0e-4ea9-ad4d-e6961af25823 None admin] PATH_INFO: `/v3/projects/8ad3c16a01344fc6a722a172049e9b0f` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:03.854909 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11fc342a-4a0e-4ea9-ad4d-e6961af25823 None admin] RBAC: Authorizing `identity:get_project(project_id=8ad3c16a01344fc6a722a172049e9b0f)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:03.856819 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11fc342a-4a0e-4ea9-ad4d-e6961af25823 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:03.859150 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 47/93] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:41:03 2022] GET /identity/v3/projects/8ad3c16a01344fc6a722a172049e9b0f => generated 312 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:41:03.865346 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11fc342a-4a0e-4ea9-ad4d-e6961af25823 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:03.867847 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:03.896810 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4db2b226-6bcb-4175-a394-ec836b8fccd4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:03.899763 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4db2b226-6bcb-4175-a394-ec836b8fccd4 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:03.977306 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11fc342a-4a0e-4ea9-ad4d-e6961af25823 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:04.027547 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b87ac884-9684-4bf4-9b3a-babc799a59f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:04.028514 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87ac884-9684-4bf4-9b3a-babc799a59f7 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:04.028986 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87ac884-9684-4bf4-9b3a-babc799a59f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:04.029489 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87ac884-9684-4bf4-9b3a-babc799a59f7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:04.030428 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b87ac884-9684-4bf4-9b3a-babc799a59f7 None admin] RBAC: Adding query filter params (role.id=40f54f5ac0994c51960ac413da8d2cc9, scope.project.id=8ad3c16a01344fc6a722a172049e9b0f, user.id=6980cb57558a43d2b384c7b561d7af70) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:04.030938 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b87ac884-9684-4bf4-9b3a-babc799a59f7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:04.032571 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b87ac884-9684-4bf4-9b3a-babc799a59f7 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:04.045291 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 48/94] 10.222.0.52 () {56 vars in 1374 bytes} [Sun Sep 25 00:41:03 2022] GET /identity/v3/role_assignments?user.id=6980cb57558a43d2b384c7b561d7af70&scope.project.id=8ad3c16a01344fc6a722a172049e9b0f&role.id=40f54f5ac0994c51960ac413da8d2cc9 => generated 259 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:41:04.205298 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 47/95] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:41:03 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 887 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 00:41:04.213845 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b87ac884-9684-4bf4-9b3a-babc799a59f7 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:04.226474 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:04.392015 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b87ac884-9684-4bf4-9b3a-babc799a59f7 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:04.476441 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7717020f-2c26-4cf7-9b68-60be71a74e11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:04.485826 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7717020f-2c26-4cf7-9b68-60be71a74e11 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:04.486182 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7717020f-2c26-4cf7-9b68-60be71a74e11 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:04.486655 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7717020f-2c26-4cf7-9b68-60be71a74e11 None admin] PATH_INFO: `/v3/projects/cd69c40698884b3192bb19ba6968842d` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:04.502854 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7717020f-2c26-4cf7-9b68-60be71a74e11 None admin] RBAC: Authorizing `identity:get_project(project_id=cd69c40698884b3192bb19ba6968842d)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:04.505806 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7717020f-2c26-4cf7-9b68-60be71a74e11 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:04.509027 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 49/96] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:41:04 2022] GET /identity/v3/projects/cd69c40698884b3192bb19ba6968842d => generated 298 bytes in 296 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:04.517944 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db2b226-6bcb-4175-a394-ec836b8fccd4 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:04.730782 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:04.746561 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db2b226-6bcb-4175-a394-ec836b8fccd4 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:04.829959 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ead684-56e1-47ea-9a27-f36ec61ab205 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:04.831024 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ead684-56e1-47ea-9a27-f36ec61ab205 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:04.831500 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ead684-56e1-47ea-9a27-f36ec61ab205 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:04.831984 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ead684-56e1-47ea-9a27-f36ec61ab205 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:04.833240 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9ead684-56e1-47ea-9a27-f36ec61ab205 None admin] RBAC: Adding query filter params (role.id=a917909d7aa84d00a841c77e7e7bdc8a, scope.project.id=cd69c40698884b3192bb19ba6968842d, user.id=6980cb57558a43d2b384c7b561d7af70) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:04.833860 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9ead684-56e1-47ea-9a27-f36ec61ab205 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:04.836411 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9ead684-56e1-47ea-9a27-f36ec61ab205 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:04.852079 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 48/97] 10.222.0.52 () {56 vars in 1374 bytes} [Sun Sep 25 00:41:04 2022] GET /identity/v3/role_assignments?user.id=6980cb57558a43d2b384c7b561d7af70&scope.project.id=cd69c40698884b3192bb19ba6968842d&role.id=a917909d7aa84d00a841c77e7e7bdc8a => generated 259 bytes in 335 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:05.797867 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf60659-7b4d-4c7b-a3fe-0229bdd7fb47 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:05.799331 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf60659-7b4d-4c7b-a3fe-0229bdd7fb47 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:05.800088 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf60659-7b4d-4c7b-a3fe-0229bdd7fb47 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:05.802831 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 50/98] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:05 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:05.809857 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9ebb00-59ff-4c03-9f1d-06f02c93ecfc None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:05.811270 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9ebb00-59ff-4c03-9f1d-06f02c93ecfc None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:05.811989 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9ebb00-59ff-4c03-9f1d-06f02c93ecfc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:05.853022 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-ae9ebb00-59ff-4c03-9f1d-06f02c93ecfc None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:05.855969 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-ae9ebb00-59ff-4c03-9f1d-06f02c93ecfc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:05.951779 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 49/99] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:41:06.375739 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0fbd8f-68d8-4e90-95a8-219627a4033d None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:06.375739 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0fbd8f-68d8-4e90-95a8-219627a4033d None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:06.375739 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0fbd8f-68d8-4e90-95a8-219627a4033d None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:06.375739 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 51/100] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:06.382583 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc60e3c-2eb7-471f-a403-d4c959210768 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:06.383125 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc60e3c-2eb7-471f-a403-d4c959210768 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:06.383584 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc60e3c-2eb7-471f-a403-d4c959210768 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:06.404080 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-6cc60e3c-2eb7-471f-a403-d4c959210768 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:06.405632 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-6cc60e3c-2eb7-471f-a403-d4c959210768 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:06.476008 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 50/101] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:41:06.493865 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e0fbd8f-68d8-4e90-95a8-219627a4033d None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:06.604173 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:06.646068 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e0fbd8f-68d8-4e90-95a8-219627a4033d None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:06.700408 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0107ec3-39dd-4329-82e5-2ef7247c7fb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:06.701074 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0107ec3-39dd-4329-82e5-2ef7247c7fb5 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:06.701381 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0107ec3-39dd-4329-82e5-2ef7247c7fb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:06.701654 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0107ec3-39dd-4329-82e5-2ef7247c7fb5 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:06.717216 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0107ec3-39dd-4329-82e5-2ef7247c7fb5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:41:06.722268 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-d0107ec3-39dd-4329-82e5-2ef7247c7fb5 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 25 00:41:06.722268 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:41:06.722268 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:41:06.722268 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:41:06.722268 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:41:06.722268 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:41:06.722268 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:41:06.722268 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:41:06.722268 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:41:06.722268 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:41:06.722268 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:41:06.722268 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:41:06.722268 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:41:06.722268 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:41:06.722268 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:41:06.722268 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:41:06.723155 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:41:06.723155 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:41:06.723155 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:41:06.723155 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:41:06.723155 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:41:06.723155 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:41:06.723155 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:41:06.723155 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:41:06.723155 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:41:06.723155 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:41:06.723155 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:41:06.723155 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:41:06.723155 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:41:06.723155 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:41:06.723155 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:41:06.723155 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:41:06.723909 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:41:06.723909 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:41:06.723909 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:41:06.723909 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:41:06.723909 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:41:06.723909 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:41:06.723909 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:41:06.723909 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:41:06.723909 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:41:06.723909 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:41:06.723909 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 25 00:41:06.723909 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:41:06.723909 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 52/102] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:41:06 2022] GET /identity/v3/roles/reader => generated 84 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:41:06.732087 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc60e3c-2eb7-471f-a403-d4c959210768 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:06.765363 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:06.859647 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc60e3c-2eb7-471f-a403-d4c959210768 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:06.897333 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfcd534-dbaf-492e-ac4d-54fa1be5daea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:06.897877 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfcd534-dbaf-492e-ac4d-54fa1be5daea None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:06.898152 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfcd534-dbaf-492e-ac4d-54fa1be5daea None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:06.898420 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfcd534-dbaf-492e-ac4d-54fa1be5daea None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:06.899087 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecfcd534-dbaf-492e-ac4d-54fa1be5daea None admin] RBAC: Adding query filter params (name=reader) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:06.899383 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecfcd534-dbaf-492e-ac4d-54fa1be5daea None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:06.901558 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecfcd534-dbaf-492e-ac4d-54fa1be5daea None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:06.906330 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0107ec3-39dd-4329-82e5-2ef7247c7fb5 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:06.910396 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 51/103] 10.222.0.52 () {56 vars in 1101 bytes} [Sun Sep 25 00:41:06 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:06.919349 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfcd534-dbaf-492e-ac4d-54fa1be5daea None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:07.013854 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:07.040251 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfcd534-dbaf-492e-ac4d-54fa1be5daea None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:07.077248 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0107ec3-39dd-4329-82e5-2ef7247c7fb5 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:07.084092 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2abe46c3-d1f8-44fe-bf79-e166ce76afb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:07.085037 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2abe46c3-d1f8-44fe-bf79-e166ce76afb4 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:07.085553 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2abe46c3-d1f8-44fe-bf79-e166ce76afb4 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:07.085882 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2abe46c3-d1f8-44fe-bf79-e166ce76afb4 None admin] PATH_INFO: `/v3/users/f079800ac9f04bf2b626bb34da78bd52` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:07.103233 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2abe46c3-d1f8-44fe-bf79-e166ce76afb4 None admin] RBAC: Authorizing `identity:get_user(user_id=f079800ac9f04bf2b626bb34da78bd52)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:07.105690 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2abe46c3-d1f8-44fe-bf79-e166ce76afb4 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:07.107692 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 52/104] 10.222.0.52 () {56 vars in 1177 bytes} [Sun Sep 25 00:41:06 2022] GET /identity/v3/users/f079800ac9f04bf2b626bb34da78bd52 => generated 289 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:07.109164 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5100622d-84db-449c-b81a-be81dbfdf7c6 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:07.109636 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5100622d-84db-449c-b81a-be81dbfdf7c6 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:07.110056 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5100622d-84db-449c-b81a-be81dbfdf7c6 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:07.111649 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 53/105] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:07.112826 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba430aa-0a85-4e2e-ab0b-485252e36530 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:07.113515 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba430aa-0a85-4e2e-ab0b-485252e36530 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:07.113910 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba430aa-0a85-4e2e-ab0b-485252e36530 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:07.115314 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 54/106] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:07.120215 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba430aa-0a85-4e2e-ab0b-485252e36530 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:07.121752 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a802cce-7866-4f1c-9225-f805694aa8b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:07.122504 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a802cce-7866-4f1c-9225-f805694aa8b7 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:07.122886 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a802cce-7866-4f1c-9225-f805694aa8b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:07.123284 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a802cce-7866-4f1c-9225-f805694aa8b7 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:07.125861 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:07.148373 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a802cce-7866-4f1c-9225-f805694aa8b7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:41:07.154308 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-9a802cce-7866-4f1c-9225-f805694aa8b7 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 25 00:41:07.154308 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:41:07.154308 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:41:07.154308 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:41:07.154308 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:41:07.154308 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:41:07.154308 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:41:07.154308 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:41:07.154308 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:41:07.154308 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:41:07.154308 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:41:07.154308 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:41:07.154308 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:41:07.154308 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:41:07.154308 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:41:07.154308 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:41:07.155200 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:41:07.155200 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:41:07.155200 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:41:07.155200 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:41:07.155200 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:41:07.155200 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:41:07.155200 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:41:07.155200 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:41:07.155200 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:41:07.155200 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:41:07.155200 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:41:07.155200 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:41:07.155200 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:41:07.155200 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:41:07.155200 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:41:07.155200 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:41:07.155968 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:41:07.155968 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:41:07.155968 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:41:07.155968 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:41:07.155968 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:41:07.155968 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:41:07.155968 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:41:07.155968 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:41:07.155968 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:41:07.155968 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:41:07.155968 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 25 00:41:07.155968 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:41:07.158332 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 53/107] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:41:06 2022] GET /identity/v3/roles/member => generated 84 bytes in 252 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:41:07.159747 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82314675-637a-4caf-96fe-4794110bf14f None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:07.160199 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82314675-637a-4caf-96fe-4794110bf14f None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:07.160629 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82314675-637a-4caf-96fe-4794110bf14f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:07.181337 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-82314675-637a-4caf-96fe-4794110bf14f None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:07.182964 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-82314675-637a-4caf-96fe-4794110bf14f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:07.256580 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 54/108] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:41:07.265420 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0ff9de-b166-4808-b6d2-bdf7b6ebcb48 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:07.266156 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0ff9de-b166-4808-b6d2-bdf7b6ebcb48 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:07.266815 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0ff9de-b166-4808-b6d2-bdf7b6ebcb48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:07.268985 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba430aa-0a85-4e2e-ab0b-485252e36530 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:07.308905 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd6e8ae-1a24-442c-86bd-d76d83611b66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:07.309689 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd6e8ae-1a24-442c-86bd-d76d83611b66 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:07.310042 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd6e8ae-1a24-442c-86bd-d76d83611b66 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:07.310375 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd6e8ae-1a24-442c-86bd-d76d83611b66 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:07.310918 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-bd0ff9de-b166-4808-b6d2-bdf7b6ebcb48 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:07.312349 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-bd0ff9de-b166-4808-b6d2-bdf7b6ebcb48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:07.314207 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:07.439571 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 55/109] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:41:07.440843 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0ff9de-b166-4808-b6d2-bdf7b6ebcb48 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:07.514092 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebd6e8ae-1a24-442c-86bd-d76d83611b66 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:07.515565 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebd6e8ae-1a24-442c-86bd-d76d83611b66 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:07.521297 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:07.589453 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0ff9de-b166-4808-b6d2-bdf7b6ebcb48 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:07.632689 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33c368a0-c6f4-45d4-b632-8b0762d47e95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:07.633626 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c368a0-c6f4-45d4-b632-8b0762d47e95 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:07.635575 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c368a0-c6f4-45d4-b632-8b0762d47e95 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:07.636064 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c368a0-c6f4-45d4-b632-8b0762d47e95 None admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:07.636915 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33c368a0-c6f4-45d4-b632-8b0762d47e95 None admin] RBAC: Adding query filter params (name=member) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:07.637225 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33c368a0-c6f4-45d4-b632-8b0762d47e95 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:07.638752 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33c368a0-c6f4-45d4-b632-8b0762d47e95 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:07.650986 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 56/110] 10.222.0.52 () {56 vars in 1101 bytes} [Sun Sep 25 00:41:07 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:07.653591 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab0d2db-e6e2-4dea-b334-613730116fb0 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:07.653879 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab0d2db-e6e2-4dea-b334-613730116fb0 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:07.654150 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab0d2db-e6e2-4dea-b334-613730116fb0 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:07.655510 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 57/111] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:07.656260 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab0d2db-e6e2-4dea-b334-613730116fb0 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:07.673540 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 55/112] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:41:07 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 554 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 00:41:07.675073 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd6e8ae-1a24-442c-86bd-d76d83611b66 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:07.714944 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:07.767633 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab0d2db-e6e2-4dea-b334-613730116fb0 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:07.785576 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:07.795282 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd6e8ae-1a24-442c-86bd-d76d83611b66 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:07.804494 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65cb122a-f205-47a3-8336-54553ab782a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:07.805063 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cb122a-f205-47a3-8336-54553ab782a6 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:07.805384 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cb122a-f205-47a3-8336-54553ab782a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:07.805660 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cb122a-f205-47a3-8336-54553ab782a6 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:07.807315 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65cb122a-f205-47a3-8336-54553ab782a6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:07.809003 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65cb122a-f205-47a3-8336-54553ab782a6 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:07.810629 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 58/113] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:41:07 2022] GET /identity/v3/domains/default => generated 199 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:07.814877 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c200472-8c7c-4d43-851f-27a3dc24dfcd None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:07.815684 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c200472-8c7c-4d43-851f-27a3dc24dfcd None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:07.815963 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c200472-8c7c-4d43-851f-27a3dc24dfcd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:07.833269 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db8403f0-803c-4b27-a6f5-a508c76c9c51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:07.834024 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8403f0-803c-4b27-a6f5-a508c76c9c51 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:07.834299 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8403f0-803c-4b27-a6f5-a508c76c9c51 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:07.834567 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8403f0-803c-4b27-a6f5-a508c76c9c51 None admin] PATH_INFO: `/v3/users/6980cb57558a43d2b384c7b561d7af70` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:07.835413 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-1c200472-8c7c-4d43-851f-27a3dc24dfcd None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:07.836601 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-1c200472-8c7c-4d43-851f-27a3dc24dfcd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:07.837185 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db8403f0-803c-4b27-a6f5-a508c76c9c51 None admin] RBAC: Authorizing `identity:get_user(user_id=6980cb57558a43d2b384c7b561d7af70)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:07.839453 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db8403f0-803c-4b27-a6f5-a508c76c9c51 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:07.841323 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 56/114] 10.222.0.52 () {56 vars in 1177 bytes} [Sun Sep 25 00:41:07 2022] GET /identity/v3/users/6980cb57558a43d2b384c7b561d7af70 => generated 275 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:07.842023 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db8403f0-803c-4b27-a6f5-a508c76c9c51 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:07.915638 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 59/115] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:41:07.917091 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c200472-8c7c-4d43-851f-27a3dc24dfcd None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:07.957525 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db8403f0-803c-4b27-a6f5-a508c76c9c51 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:07.961152 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:08.001804 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cb25d07-3a66-4987-a8be-7ce732001ced None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:08.002488 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb25d07-3a66-4987-a8be-7ce732001ced None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:08.002768 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb25d07-3a66-4987-a8be-7ce732001ced None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:08.003104 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb25d07-3a66-4987-a8be-7ce732001ced None admin] PATH_INFO: `/v3/projects/cd69c40698884b3192bb19ba6968842d` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:08.004407 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cb25d07-3a66-4987-a8be-7ce732001ced None admin] RBAC: Authorizing `identity:get_project(project_id=cd69c40698884b3192bb19ba6968842d)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:08.006056 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cb25d07-3a66-4987-a8be-7ce732001ced None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:08.007893 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 57/116] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:41:07 2022] GET /identity/v3/projects/cd69c40698884b3192bb19ba6968842d => generated 298 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:08.008715 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cb25d07-3a66-4987-a8be-7ce732001ced None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:08.063476 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c200472-8c7c-4d43-851f-27a3dc24dfcd None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:08.082881 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:08.112521 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2faf4bf9-ed09-40c8-8730-adbc8785baa6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:08.113861 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2faf4bf9-ed09-40c8-8730-adbc8785baa6 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:08.114346 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2faf4bf9-ed09-40c8-8730-adbc8785baa6 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:08.114830 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2faf4bf9-ed09-40c8-8730-adbc8785baa6 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:08.127220 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2faf4bf9-ed09-40c8-8730-adbc8785baa6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:41:08.132728 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-2faf4bf9-ed09-40c8-8730-adbc8785baa6 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:41:08.132728 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:41:08.132728 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:41:08.132728 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:41:08.132728 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:41:08.132728 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:41:08.132728 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:41:08.132728 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:41:08.132728 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:41:08.132728 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:41:08.132728 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:41:08.132728 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:41:08.132728 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:41:08.132728 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:41:08.132728 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:41:08.132728 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:41:08.133626 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:41:08.133626 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:41:08.133626 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:41:08.133626 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:41:08.133626 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:41:08.133626 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:41:08.133626 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:41:08.133626 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:41:08.133626 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:41:08.133626 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:41:08.133626 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:41:08.133626 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:41:08.133626 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:41:08.133626 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:41:08.133626 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:41:08.133626 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:41:08.134493 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:41:08.134493 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:41:08.134493 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:41:08.134493 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:41:08.134493 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:41:08.134493 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:41:08.134493 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:41:08.134493 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:41:08.134493 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:41:08.134493 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:41:08.134493 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:41:08.134493 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:41:08.137646 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 60/117] 10.222.0.52 () {56 vars in 1096 bytes} [Sun Sep 25 00:41:07 2022] GET /identity/v3/roles/admin => generated 83 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:41:08.138696 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2faf4bf9-ed09-40c8-8730-adbc8785baa6 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:08.141865 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cb25d07-3a66-4987-a8be-7ce732001ced None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:08.180180 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f359dafc-d3dc-4142-9fe0-25455ebf4fcf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:08.180788 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f359dafc-d3dc-4142-9fe0-25455ebf4fcf None admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:08.181071 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f359dafc-d3dc-4142-9fe0-25455ebf4fcf None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:08.181394 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f359dafc-d3dc-4142-9fe0-25455ebf4fcf None admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:08.182640 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f359dafc-d3dc-4142-9fe0-25455ebf4fcf None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:08.184319 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f359dafc-d3dc-4142-9fe0-25455ebf4fcf None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:08.224239 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 58/118] 10.222.0.52 () {60 vars in 1141 bytes} [Sun Sep 25 00:41:08 2022] POST /identity/v3/projects => generated 302 bytes in 207 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:41:08.224239 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f359dafc-d3dc-4142-9fe0-25455ebf4fcf None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:08.257951 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:08.271413 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2faf4bf9-ed09-40c8-8730-adbc8785baa6 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:08.316766 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d546bf6-c3a8-4f2a-ab28-0e46d08f2b7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:08.318092 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d546bf6-c3a8-4f2a-ab28-0e46d08f2b7e None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:08.318584 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d546bf6-c3a8-4f2a-ab28-0e46d08f2b7e None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:08.319044 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d546bf6-c3a8-4f2a-ab28-0e46d08f2b7e None admin] PATH_INFO: `/v3/projects/cd69c40698884b3192bb19ba6968842d` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:08.320786 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d546bf6-c3a8-4f2a-ab28-0e46d08f2b7e None admin] RBAC: Authorizing `identity:get_project(project_id=cd69c40698884b3192bb19ba6968842d)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:08.323810 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d546bf6-c3a8-4f2a-ab28-0e46d08f2b7e None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:08.326375 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 61/119] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:41:08 2022] GET /identity/v3/projects/cd69c40698884b3192bb19ba6968842d => generated 298 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:08.328867 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198572ca-a157-45d4-b498-4cb984188986 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:08.329464 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198572ca-a157-45d4-b498-4cb984188986 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:08.329939 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198572ca-a157-45d4-b498-4cb984188986 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:08.331564 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 62/120] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:08.336554 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-198572ca-a157-45d4-b498-4cb984188986 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:08.377927 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f359dafc-d3dc-4142-9fe0-25455ebf4fcf None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:08.403510 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:08.423721 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a3958e-0c72-4105-9c54-907b1bde0e07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:08.424578 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a3958e-0c72-4105-9c54-907b1bde0e07 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:08.425296 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a3958e-0c72-4105-9c54-907b1bde0e07 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:08.425296 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a3958e-0c72-4105-9c54-907b1bde0e07 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:08.425876 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92a3958e-0c72-4105-9c54-907b1bde0e07 None admin] RBAC: Adding query filter params (role.id=cc65bfe8ed2740b69427f705add475a5, scope.project.id=cd69c40698884b3192bb19ba6968842d, user.id=f079800ac9f04bf2b626bb34da78bd52) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:08.426182 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92a3958e-0c72-4105-9c54-907b1bde0e07 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:08.427626 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92a3958e-0c72-4105-9c54-907b1bde0e07 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:08.434936 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 59/121] 10.222.0.52 () {56 vars in 1374 bytes} [Sun Sep 25 00:41:08 2022] GET /identity/v3/role_assignments?user.id=f079800ac9f04bf2b626bb34da78bd52&scope.project.id=cd69c40698884b3192bb19ba6968842d&role.id=cc65bfe8ed2740b69427f705add475a5 => generated 259 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:08.436389 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a3958e-0c72-4105-9c54-907b1bde0e07 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:08.482655 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-198572ca-a157-45d4-b498-4cb984188986 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:08.530924 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09613684-3ed3-429d-879b-d8aba63a9ecb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:08.531460 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09613684-3ed3-429d-879b-d8aba63a9ecb None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:08.531802 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09613684-3ed3-429d-879b-d8aba63a9ecb None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:08.532078 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09613684-3ed3-429d-879b-d8aba63a9ecb None admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:08.532828 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09613684-3ed3-429d-879b-d8aba63a9ecb None admin] RBAC: Adding query filter params (name=admin) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:08.533167 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09613684-3ed3-429d-879b-d8aba63a9ecb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:08.534621 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09613684-3ed3-429d-879b-d8aba63a9ecb None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:08.545240 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 63/122] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:41:08 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:08.546157 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09613684-3ed3-429d-879b-d8aba63a9ecb None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:08.557931 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:08.576501 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a3958e-0c72-4105-9c54-907b1bde0e07 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:08.628558 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b624fa0-22f3-4941-9a44-83a8e158e394 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:08.629197 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b624fa0-22f3-4941-9a44-83a8e158e394 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:08.630857 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b624fa0-22f3-4941-9a44-83a8e158e394 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:08.631140 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b624fa0-22f3-4941-9a44-83a8e158e394 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:08.646789 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b624fa0-22f3-4941-9a44-83a8e158e394 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:41:08.654832 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-3b624fa0-22f3-4941-9a44-83a8e158e394 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 25 00:41:08.654832 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:41:08.654832 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:41:08.654832 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:41:08.654832 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:41:08.654832 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:41:08.654832 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:41:08.654832 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:41:08.654832 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:41:08.654832 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:41:08.654832 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:41:08.654832 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:41:08.654832 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:41:08.654832 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:41:08.654832 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:41:08.654832 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:41:08.655718 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:41:08.655718 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:41:08.655718 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:41:08.655718 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:41:08.655718 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:41:08.655718 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:41:08.655718 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:41:08.655718 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:41:08.655718 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:41:08.655718 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:41:08.655718 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:41:08.655718 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:41:08.655718 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:41:08.655718 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:41:08.655718 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:41:08.655718 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:41:08.656486 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:41:08.656486 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:41:08.656486 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:41:08.656486 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:41:08.656486 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:41:08.656486 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:41:08.656486 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:41:08.656486 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:41:08.656486 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:41:08.656486 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:41:08.656486 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 25 00:41:08.656486 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:41:08.660399 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 60/123] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:41:08 2022] GET /identity/v3/roles/member => generated 84 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:41:08.662860 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b30b4a4-dab2-46b9-9d73-dcd0ed558581 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:08.663406 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b30b4a4-dab2-46b9-9d73-dcd0ed558581 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:08.664392 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b30b4a4-dab2-46b9-9d73-dcd0ed558581 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:08.688104 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-0b30b4a4-dab2-46b9-9d73-dcd0ed558581 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:08.689691 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-0b30b4a4-dab2-46b9-9d73-dcd0ed558581 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:08.690369 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09613684-3ed3-429d-879b-d8aba63a9ecb None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:08.693777 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:08.731179 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee1e4fa-25ce-4474-aa14-a8b572d883d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:08.731891 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee1e4fa-25ce-4474-aa14-a8b572d883d5 None admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:08.732267 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee1e4fa-25ce-4474-aa14-a8b572d883d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:08.732436 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee1e4fa-25ce-4474-aa14-a8b572d883d5 None admin] PATH_INFO: `/v3/projects/cd69c40698884b3192bb19ba6968842d/users/6980cb57558a43d2b384c7b561d7af70/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:08.734730 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ee1e4fa-25ce-4474-aa14-a8b572d883d5 None admin] RBAC: Authorizing `identity:create_grant(project_id=cd69c40698884b3192bb19ba6968842d, user_id=6980cb57558a43d2b384c7b561d7af70, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:08.736685 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ee1e4fa-25ce-4474-aa14-a8b572d883d5 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:08.752708 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 64/124] 10.222.0.52 () {58 vars in 1439 bytes} [Sun Sep 25 00:41:08 2022] PUT /identity/v3/projects/cd69c40698884b3192bb19ba6968842d/users/6980cb57558a43d2b384c7b561d7af70/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:41:08.753574 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee1e4fa-25ce-4474-aa14-a8b572d883d5 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:08.780481 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 61/125] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:41:08.781822 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b30b4a4-dab2-46b9-9d73-dcd0ed558581 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:08.874563 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:08.900732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b30b4a4-dab2-46b9-9d73-dcd0ed558581 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:08.919675 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee1e4fa-25ce-4474-aa14-a8b572d883d5 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:08.940980 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f490bf-b8eb-4ce7-a665-b59d1d3e8dd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:08.942078 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f490bf-b8eb-4ce7-a665-b59d1d3e8dd7 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:08.942560 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f490bf-b8eb-4ce7-a665-b59d1d3e8dd7 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:08.943077 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f490bf-b8eb-4ce7-a665-b59d1d3e8dd7 None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:08.943973 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8f490bf-b8eb-4ce7-a665-b59d1d3e8dd7 None admin] RBAC: Adding query filter params (name=member) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:08.944471 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8f490bf-b8eb-4ce7-a665-b59d1d3e8dd7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:08.946116 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8f490bf-b8eb-4ce7-a665-b59d1d3e8dd7 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:08.956794 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 62/126] 10.222.0.52 () {56 vars in 1101 bytes} [Sun Sep 25 00:41:08 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:08.968179 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f490bf-b8eb-4ce7-a665-b59d1d3e8dd7 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:08.992937 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d6c6fd-c711-4faa-affa-1a50a9ba58f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:08.994060 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d6c6fd-c711-4faa-affa-1a50a9ba58f1 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:08.994359 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d6c6fd-c711-4faa-affa-1a50a9ba58f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:08.995351 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d6c6fd-c711-4faa-affa-1a50a9ba58f1 None admin] PATH_INFO: `/v3/users/d6d73e23626043529ac5b9e3889933d4` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:08.996781 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40d6c6fd-c711-4faa-affa-1a50a9ba58f1 None admin] RBAC: Authorizing `identity:get_user(user_id=d6d73e23626043529ac5b9e3889933d4)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:09.000884 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40d6c6fd-c711-4faa-affa-1a50a9ba58f1 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:09.003040 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 65/127] 10.222.0.52 () {56 vars in 1177 bytes} [Sun Sep 25 00:41:08 2022] GET /identity/v3/users/d6d73e23626043529ac5b9e3889933d4 => generated 247 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:09.008666 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d6c6fd-c711-4faa-affa-1a50a9ba58f1 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:09.066702 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:09.196180 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f490bf-b8eb-4ce7-a665-b59d1d3e8dd7 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:09.200954 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d6c6fd-c711-4faa-affa-1a50a9ba58f1 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:09.258879 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae01e22-12d7-473e-80ce-2183af4a4413 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:09.259522 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae01e22-12d7-473e-80ce-2183af4a4413 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:09.259862 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae01e22-12d7-473e-80ce-2183af4a4413 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:09.260138 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae01e22-12d7-473e-80ce-2183af4a4413 None admin] PATH_INFO: `/v3/users/6980cb57558a43d2b384c7b561d7af70` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:09.261493 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aae01e22-12d7-473e-80ce-2183af4a4413 None admin] RBAC: Authorizing `identity:get_user(user_id=6980cb57558a43d2b384c7b561d7af70)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:09.262948 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aae01e22-12d7-473e-80ce-2183af4a4413 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:09.269495 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d6b44dc-84f4-41d8-a321-3c757510a524 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:09.270135 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6b44dc-84f4-41d8-a321-3c757510a524 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:09.270414 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6b44dc-84f4-41d8-a321-3c757510a524 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:09.270686 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6b44dc-84f4-41d8-a321-3c757510a524 None admin] PATH_INFO: `/v3/projects/cd69c40698884b3192bb19ba6968842d` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:09.272090 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 63/128] 10.222.0.52 () {56 vars in 1177 bytes} [Sun Sep 25 00:41:08 2022] GET /identity/v3/users/6980cb57558a43d2b384c7b561d7af70 => generated 275 bytes in 306 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:09.273270 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d6b44dc-84f4-41d8-a321-3c757510a524 None admin] RBAC: Authorizing `identity:get_project(project_id=cd69c40698884b3192bb19ba6968842d)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:09.274801 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d6b44dc-84f4-41d8-a321-3c757510a524 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:09.277872 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae01e22-12d7-473e-80ce-2183af4a4413 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:09.280670 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 66/129] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:41:09 2022] GET /identity/v3/projects/cd69c40698884b3192bb19ba6968842d => generated 298 bytes in 272 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:41:09.290609 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d6b44dc-84f4-41d8-a321-3c757510a524 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:09.295460 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:09.423401 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae01e22-12d7-473e-80ce-2183af4a4413 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:09.423766 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:09.427434 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d6b44dc-84f4-41d8-a321-3c757510a524 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:09.480406 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2904711-2890-4261-b3d2-635e28b1e295 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:09.481930 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2904711-2890-4261-b3d2-635e28b1e295 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:09.482880 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2904711-2890-4261-b3d2-635e28b1e295 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:09.483817 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2904711-2890-4261-b3d2-635e28b1e295 None admin] PATH_INFO: `/v3/projects/8ad3c16a01344fc6a722a172049e9b0f` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:09.486459 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2904711-2890-4261-b3d2-635e28b1e295 None admin] RBAC: Authorizing `identity:get_project(project_id=8ad3c16a01344fc6a722a172049e9b0f)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:09.489293 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2904711-2890-4261-b3d2-635e28b1e295 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:09.493281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 64/130] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:41:09 2022] GET /identity/v3/projects/8ad3c16a01344fc6a722a172049e9b0f => generated 312 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:41:09.493281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2904711-2890-4261-b3d2-635e28b1e295 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:09.493506 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33b1313-bd95-4b2a-812b-0d9d8b4671ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:09.493506 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33b1313-bd95-4b2a-812b-0d9d8b4671ff None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:09.493506 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33b1313-bd95-4b2a-812b-0d9d8b4671ff None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:09.493506 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33b1313-bd95-4b2a-812b-0d9d8b4671ff None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:09.519426 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e33b1313-bd95-4b2a-812b-0d9d8b4671ff None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:41:09.525140 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-e33b1313-bd95-4b2a-812b-0d9d8b4671ff None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 00:41:09.525140 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:41:09.525140 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:41:09.525140 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:41:09.525140 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:41:09.525140 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:41:09.525140 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:41:09.525140 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:41:09.525140 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:41:09.525140 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:41:09.525140 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:41:09.525140 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:41:09.525140 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:41:09.525140 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:41:09.525140 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:41:09.525140 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:41:09.526358 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:41:09.526358 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:41:09.526358 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:41:09.526358 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:41:09.526358 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:41:09.526358 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:41:09.526358 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:41:09.526358 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:41:09.526358 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:41:09.526358 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:41:09.526358 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:41:09.526358 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:41:09.526358 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:41:09.526358 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:41:09.526358 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:41:09.526358 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:41:09.527226 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:41:09.527226 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:41:09.527226 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:41:09.527226 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:41:09.527226 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:41:09.527226 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:41:09.527226 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:41:09.527226 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:41:09.527226 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:41:09.527226 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:41:09.527226 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 00:41:09.527226 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:41:09.527226 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 67/131] 10.222.0.52 () {56 vars in 1114 bytes} [Sun Sep 25 00:41:09 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:41:09.533933 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33b1313-bd95-4b2a-812b-0d9d8b4671ff None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:09.597623 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:09.645090 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2904711-2890-4261-b3d2-635e28b1e295 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:09.708983 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b1f055d-1016-40aa-88f1-73efb99f616e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:09.709599 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1f055d-1016-40aa-88f1-73efb99f616e None admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:09.709873 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1f055d-1016-40aa-88f1-73efb99f616e None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:09.710141 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1f055d-1016-40aa-88f1-73efb99f616e None admin] PATH_INFO: `/v3/projects/cd69c40698884b3192bb19ba6968842d/users/d6d73e23626043529ac5b9e3889933d4/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:09.712090 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b1f055d-1016-40aa-88f1-73efb99f616e None admin] RBAC: Authorizing `identity:create_grant(project_id=cd69c40698884b3192bb19ba6968842d, user_id=d6d73e23626043529ac5b9e3889933d4, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:09.713932 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b1f055d-1016-40aa-88f1-73efb99f616e None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:09.729175 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33b1313-bd95-4b2a-812b-0d9d8b4671ff None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:09.737026 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 65/132] 10.222.0.52 () {58 vars in 1439 bytes} [Sun Sep 25 00:41:09 2022] PUT /identity/v3/projects/cd69c40698884b3192bb19ba6968842d/users/d6d73e23626043529ac5b9e3889933d4/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:41:09.737897 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b1f055d-1016-40aa-88f1-73efb99f616e None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:09.808838 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfc83910-8f53-4238-beb2-9db08158197e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:09.809722 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc83910-8f53-4238-beb2-9db08158197e None admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:09.810003 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc83910-8f53-4238-beb2-9db08158197e None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:09.810278 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc83910-8f53-4238-beb2-9db08158197e None admin] PATH_INFO: `/v3/projects/8ad3c16a01344fc6a722a172049e9b0f/users/6980cb57558a43d2b384c7b561d7af70/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:09.812498 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfc83910-8f53-4238-beb2-9db08158197e None admin] RBAC: Authorizing `identity:create_grant(project_id=8ad3c16a01344fc6a722a172049e9b0f, user_id=6980cb57558a43d2b384c7b561d7af70, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:09.814578 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfc83910-8f53-4238-beb2-9db08158197e None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:09.840122 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 68/133] 10.222.0.52 () {58 vars in 1439 bytes} [Sun Sep 25 00:41:09 2022] PUT /identity/v3/projects/8ad3c16a01344fc6a722a172049e9b0f/users/6980cb57558a43d2b384c7b561d7af70/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 313 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:41:09.958280 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b1f055d-1016-40aa-88f1-73efb99f616e None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:10.011179 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f30715-8d34-44a4-ac63-0e236231d800 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:10.012616 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f30715-8d34-44a4-ac63-0e236231d800 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:10.013097 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f30715-8d34-44a4-ac63-0e236231d800 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:10.013616 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f30715-8d34-44a4-ac63-0e236231d800 None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:10.014503 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3f30715-8d34-44a4-ac63-0e236231d800 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:10.015002 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3f30715-8d34-44a4-ac63-0e236231d800 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:10.016627 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3f30715-8d34-44a4-ac63-0e236231d800 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:10.022551 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:10.030776 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 66/134] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:41:09 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 293 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:10.036665 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfc83910-8f53-4238-beb2-9db08158197e None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:10.082439 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:10.201923 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfc83910-8f53-4238-beb2-9db08158197e None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:10.240712 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e9cdca-1734-45ff-8ee0-35288cc78899 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:10.241386 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e9cdca-1734-45ff-8ee0-35288cc78899 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:10.241670 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e9cdca-1734-45ff-8ee0-35288cc78899 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:10.241945 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e9cdca-1734-45ff-8ee0-35288cc78899 None admin] PATH_INFO: `/v3/users/6980cb57558a43d2b384c7b561d7af70` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:10.243090 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e9cdca-1734-45ff-8ee0-35288cc78899 None admin] RBAC: Authorizing `identity:get_user(user_id=6980cb57558a43d2b384c7b561d7af70)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:10.244564 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e9cdca-1734-45ff-8ee0-35288cc78899 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:10.246226 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 69/135] 10.222.0.52 () {56 vars in 1177 bytes} [Sun Sep 25 00:41:10 2022] GET /identity/v3/users/6980cb57558a43d2b384c7b561d7af70 => generated 275 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:10.252111 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f30715-8d34-44a4-ac63-0e236231d800 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:10.396537 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f30715-8d34-44a4-ac63-0e236231d800 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:10.433603 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3a5722f-73df-45b1-a897-a8a0f16bb305 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:10.434254 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a5722f-73df-45b1-a897-a8a0f16bb305 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:10.434529 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a5722f-73df-45b1-a897-a8a0f16bb305 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:10.434799 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a5722f-73df-45b1-a897-a8a0f16bb305 None admin] PATH_INFO: `/v3/projects/cd69c40698884b3192bb19ba6968842d` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:10.435973 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3a5722f-73df-45b1-a897-a8a0f16bb305 None admin] RBAC: Authorizing `identity:get_project(project_id=cd69c40698884b3192bb19ba6968842d)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:10.437470 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3a5722f-73df-45b1-a897-a8a0f16bb305 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:10.439491 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 67/136] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:41:10 2022] GET /identity/v3/projects/cd69c40698884b3192bb19ba6968842d => generated 298 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:10.445207 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e9cdca-1734-45ff-8ee0-35288cc78899 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:10.601033 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:10.623029 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e9cdca-1734-45ff-8ee0-35288cc78899 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:10.682260 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e41bedab-c0c2-485c-bc86-07218ec931f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:10.682867 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41bedab-c0c2-485c-bc86-07218ec931f3 None admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:10.683152 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41bedab-c0c2-485c-bc86-07218ec931f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:10.683433 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41bedab-c0c2-485c-bc86-07218ec931f3 None admin] PATH_INFO: `/v3/projects/cd69c40698884b3192bb19ba6968842d/users/6980cb57558a43d2b384c7b561d7af70/roles/a917909d7aa84d00a841c77e7e7bdc8a` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:10.685772 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e41bedab-c0c2-485c-bc86-07218ec931f3 None admin] RBAC: Authorizing `identity:create_grant(project_id=cd69c40698884b3192bb19ba6968842d, user_id=6980cb57558a43d2b384c7b561d7af70, role_id=a917909d7aa84d00a841c77e7e7bdc8a)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:10.687255 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e41bedab-c0c2-485c-bc86-07218ec931f3 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:10.704533 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 70/137] 10.222.0.52 () {58 vars in 1439 bytes} [Sun Sep 25 00:41:10 2022] PUT /identity/v3/projects/cd69c40698884b3192bb19ba6968842d/users/6980cb57558a43d2b384c7b561d7af70/roles/a917909d7aa84d00a841c77e7e7bdc8a => generated 0 bytes in 260 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:41:11.221704 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3deb81dd-0857-486e-b108-e7e945385c94 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:11.222717 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3deb81dd-0857-486e-b108-e7e945385c94 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:11.223209 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3deb81dd-0857-486e-b108-e7e945385c94 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:11.225043 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 68/138] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:11 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:11.231063 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64175dbb-d8ed-4fa4-a91e-b3460de78ab6 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:11.231559 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64175dbb-d8ed-4fa4-a91e-b3460de78ab6 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:11.231665 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64175dbb-d8ed-4fa4-a91e-b3460de78ab6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:11.258282 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-64175dbb-d8ed-4fa4-a91e-b3460de78ab6 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:11.259572 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-64175dbb-d8ed-4fa4-a91e-b3460de78ab6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:11.348241 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 71/139] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:41:11.482983 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc7ce6c-3501-4c53-97c9-39ed64679b36 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:11.483480 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc7ce6c-3501-4c53-97c9-39ed64679b36 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:11.483949 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc7ce6c-3501-4c53-97c9-39ed64679b36 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:11.485552 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 69/140] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:11 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:11.491588 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a57e8f-70ed-4cce-97e5-b459c087ad3b None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:11.491882 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a57e8f-70ed-4cce-97e5-b459c087ad3b None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:11.492221 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a57e8f-70ed-4cce-97e5-b459c087ad3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:11.522322 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-d1a57e8f-70ed-4cce-97e5-b459c087ad3b None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:11.523528 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-d1a57e8f-70ed-4cce-97e5-b459c087ad3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:11.618957 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 72/141] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 25 00:41:11.692063 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc7ce6c-3501-4c53-97c9-39ed64679b36 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:11.778198 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:11.826137 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc7ce6c-3501-4c53-97c9-39ed64679b36 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:11.862819 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c200b5-139b-4aeb-b08c-f8f0653eb8d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:11.863761 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c200b5-139b-4aeb-b08c-f8f0653eb8d8 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:11.864038 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c200b5-139b-4aeb-b08c-f8f0653eb8d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:11.864302 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c200b5-139b-4aeb-b08c-f8f0653eb8d8 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:11.865443 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c200b5-139b-4aeb-b08c-f8f0653eb8d8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:11.866950 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c200b5-139b-4aeb-b08c-f8f0653eb8d8 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:11.868686 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 70/142] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:41:11 2022] GET /identity/v3/domains/default => generated 199 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:11.874339 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a57e8f-70ed-4cce-97e5-b459c087ad3b None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:11.886422 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:12.003019 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a57e8f-70ed-4cce-97e5-b459c087ad3b None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:12.048519 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97a86d5-7a7a-4fd9-ac8e-407bedfc09e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:12.049048 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97a86d5-7a7a-4fd9-ac8e-407bedfc09e1 None admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:12.049341 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97a86d5-7a7a-4fd9-ac8e-407bedfc09e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:12.049623 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97a86d5-7a7a-4fd9-ac8e-407bedfc09e1 None admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:12.050282 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f97a86d5-7a7a-4fd9-ac8e-407bedfc09e1 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:12.051702 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f97a86d5-7a7a-4fd9-ac8e-407bedfc09e1 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:12.092820 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 73/143] 10.222.0.52 () {60 vars in 1132 bytes} [Sun Sep 25 00:41:11 2022] POST /identity/v3/users => generated 283 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:41:12.147071 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c200b5-139b-4aeb-b08c-f8f0653eb8d8 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:12.282148 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c200b5-139b-4aeb-b08c-f8f0653eb8d8 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:12.331623 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a3e6a4-f31d-49d1-a9a9-357bccb7d5d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:12.332606 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a3e6a4-f31d-49d1-a9a9-357bccb7d5d5 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:12.333095 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a3e6a4-f31d-49d1-a9a9-357bccb7d5d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:12.333608 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a3e6a4-f31d-49d1-a9a9-357bccb7d5d5 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:12.339782 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:12.351466 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7a3e6a4-f31d-49d1-a9a9-357bccb7d5d5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:41:12.357040 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-b7a3e6a4-f31d-49d1-a9a9-357bccb7d5d5 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 25 00:41:12.357040 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:41:12.357040 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:41:12.357040 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:41:12.357040 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:41:12.357040 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:41:12.357040 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:41:12.357040 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:41:12.357040 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:41:12.357040 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:41:12.357040 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:41:12.357040 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:41:12.357040 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:41:12.357040 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:41:12.357040 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:41:12.357040 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:41:12.357999 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:41:12.357999 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:41:12.357999 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:41:12.357999 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:41:12.357999 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:41:12.357999 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:41:12.357999 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:41:12.357999 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:41:12.357999 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:41:12.357999 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:41:12.357999 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:41:12.357999 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:41:12.357999 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:41:12.357999 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:41:12.357999 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:41:12.357999 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:41:12.358805 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:41:12.358805 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:41:12.358805 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:41:12.358805 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:41:12.358805 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:41:12.358805 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:41:12.358805 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:41:12.358805 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:41:12.358805 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:41:12.358805 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:41:12.358805 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 25 00:41:12.358805 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:41:12.365374 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 71/144] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:41:12 2022] GET /identity/v3/roles/reader => generated 84 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:41:12.368036 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97a86d5-7a7a-4fd9-ac8e-407bedfc09e1 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:12.468244 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:12.533088 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97a86d5-7a7a-4fd9-ac8e-407bedfc09e1 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:12.587278 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfc6489a-04db-4896-aa8e-890178b7d459 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:12.588476 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc6489a-04db-4896-aa8e-890178b7d459 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:12.588870 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc6489a-04db-4896-aa8e-890178b7d459 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:12.589144 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc6489a-04db-4896-aa8e-890178b7d459 None admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:12.589874 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfc6489a-04db-4896-aa8e-890178b7d459 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:12.590177 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfc6489a-04db-4896-aa8e-890178b7d459 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:12.591611 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfc6489a-04db-4896-aa8e-890178b7d459 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:12.606464 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 74/145] 10.222.0.52 () {56 vars in 1101 bytes} [Sun Sep 25 00:41:12 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:12.612612 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a3e6a4-f31d-49d1-a9a9-357bccb7d5d5 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:12.737888 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a3e6a4-f31d-49d1-a9a9-357bccb7d5d5 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:12.763540 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5adff6bc-b990-4d1e-a7ec-e4fb6daa563a None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:12.763902 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5adff6bc-b990-4d1e-a7ec-e4fb6daa563a None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:12.764273 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5adff6bc-b990-4d1e-a7ec-e4fb6daa563a None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:12.765645 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 75/146] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:12.782739 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ed3f4ce-513c-40d1-939b-8bcd80926f99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:12.782947 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb398bf-25b7-4831-9c8b-8e801acff455 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:12.783193 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb398bf-25b7-4831-9c8b-8e801acff455 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:12.783366 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed3f4ce-513c-40d1-939b-8bcd80926f99 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:12.783458 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb398bf-25b7-4831-9c8b-8e801acff455 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:12.783638 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed3f4ce-513c-40d1-939b-8bcd80926f99 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:12.783907 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed3f4ce-513c-40d1-939b-8bcd80926f99 None admin] PATH_INFO: `/v3/users/f079800ac9f04bf2b626bb34da78bd52` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:12.785042 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ed3f4ce-513c-40d1-939b-8bcd80926f99 None admin] RBAC: Authorizing `identity:get_user(user_id=f079800ac9f04bf2b626bb34da78bd52)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:12.786506 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ed3f4ce-513c-40d1-939b-8bcd80926f99 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:12.792751 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 72/147] 10.222.0.52 () {56 vars in 1177 bytes} [Sun Sep 25 00:41:12 2022] GET /identity/v3/users/f079800ac9f04bf2b626bb34da78bd52 => generated 289 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:12.794371 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ed3f4ce-513c-40d1-939b-8bcd80926f99 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:12.815528 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-2bb398bf-25b7-4831-9c8b-8e801acff455 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:12.816973 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-2bb398bf-25b7-4831-9c8b-8e801acff455 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:12.903444 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:12.950895 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 76/148] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:41:12.974072 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ed3f4ce-513c-40d1-939b-8bcd80926f99 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:13.016147 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc5dcd73-398b-437f-84ac-47aa111cb753 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:13.016793 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5dcd73-398b-437f-84ac-47aa111cb753 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:13.017068 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5dcd73-398b-437f-84ac-47aa111cb753 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:13.017367 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5dcd73-398b-437f-84ac-47aa111cb753 None admin] PATH_INFO: `/v3/projects/cd69c40698884b3192bb19ba6968842d` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:13.018539 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc5dcd73-398b-437f-84ac-47aa111cb753 None admin] RBAC: Authorizing `identity:get_project(project_id=cd69c40698884b3192bb19ba6968842d)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:13.020003 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc5dcd73-398b-437f-84ac-47aa111cb753 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:13.023594 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 73/149] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:41:12 2022] GET /identity/v3/projects/cd69c40698884b3192bb19ba6968842d => generated 298 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:41:13.030377 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb398bf-25b7-4831-9c8b-8e801acff455 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:13.104051 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:13.159245 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb398bf-25b7-4831-9c8b-8e801acff455 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:13.198414 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f8e427-6358-47c2-80e4-a2a365f12d49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:13.199119 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f8e427-6358-47c2-80e4-a2a365f12d49 None admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:13.199371 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f8e427-6358-47c2-80e4-a2a365f12d49 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:13.199641 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f8e427-6358-47c2-80e4-a2a365f12d49 None admin] PATH_INFO: `/v3/projects/cd69c40698884b3192bb19ba6968842d/users/f079800ac9f04bf2b626bb34da78bd52/roles/cc65bfe8ed2740b69427f705add475a5` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:13.201668 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49f8e427-6358-47c2-80e4-a2a365f12d49 None admin] RBAC: Authorizing `identity:create_grant(project_id=cd69c40698884b3192bb19ba6968842d, user_id=f079800ac9f04bf2b626bb34da78bd52, role_id=cc65bfe8ed2740b69427f705add475a5)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:13.203136 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49f8e427-6358-47c2-80e4-a2a365f12d49 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:13.206043 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82430199-6a4e-4fe6-b664-06552a9301c4 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:13.206323 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82430199-6a4e-4fe6-b664-06552a9301c4 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:13.206594 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82430199-6a4e-4fe6-b664-06552a9301c4 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:13.207987 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 74/150] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 00:41:13.215618 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99e4d08-0c15-4d61-a972-a119ccf00d52 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:13.215937 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99e4d08-0c15-4d61-a972-a119ccf00d52 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:13.216261 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99e4d08-0c15-4d61-a972-a119ccf00d52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:13.236586 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 77/151] 10.222.0.52 () {58 vars in 1439 bytes} [Sun Sep 25 00:41:13 2022] PUT /identity/v3/projects/cd69c40698884b3192bb19ba6968842d/users/f079800ac9f04bf2b626bb34da78bd52/roles/cc65bfe8ed2740b69427f705add475a5 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:41:13.242467 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-f99e4d08-0c15-4d61-a972-a119ccf00d52 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:13.243704 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-f99e4d08-0c15-4d61-a972-a119ccf00d52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:13.323935 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 75/152] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:41:13.599810 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f8e427-6358-47c2-80e4-a2a365f12d49 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:13.617914 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24385fe6-62db-4631-a320-bc0cc843bc0a None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:13.618607 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24385fe6-62db-4631-a320-bc0cc843bc0a None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:13.619650 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24385fe6-62db-4631-a320-bc0cc843bc0a None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:13.621672 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 76/153] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:13 2022] GET /identity => generated 270 bytes in 14 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:13.627760 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08332972-d429-459c-9c13-cf71b9ece52d None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:13.628358 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08332972-d429-459c-9c13-cf71b9ece52d None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:13.628825 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08332972-d429-459c-9c13-cf71b9ece52d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:13.669021 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-08332972-d429-459c-9c13-cf71b9ece52d None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:13.670558 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-08332972-d429-459c-9c13-cf71b9ece52d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:13.731719 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:13.744916 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 77/154] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 25 00:41:13.746205 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08332972-d429-459c-9c13-cf71b9ece52d None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:13.857996 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f8e427-6358-47c2-80e4-a2a365f12d49 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:13.889634 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:13.926564 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6999a2-8107-48a1-9aed-9fc0e4956c8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:13.927202 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6999a2-8107-48a1-9aed-9fc0e4956c8a None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:13.927480 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6999a2-8107-48a1-9aed-9fc0e4956c8a None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:13.927749 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6999a2-8107-48a1-9aed-9fc0e4956c8a None admin] PATH_INFO: `/v3/roles/member` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:13.942693 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08332972-d429-459c-9c13-cf71b9ece52d None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:13.948428 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca6999a2-8107-48a1-9aed-9fc0e4956c8a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:41:13.958783 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-ca6999a2-8107-48a1-9aed-9fc0e4956c8a None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 25 00:41:13.958783 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:41:13.958783 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:41:13.958783 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:41:13.958783 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:41:13.958783 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:41:13.958783 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:41:13.958783 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:41:13.958783 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:41:13.958783 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:41:13.958783 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:41:13.958783 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:41:13.958783 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:41:13.958783 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:41:13.958783 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:41:13.958783 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:41:13.959681 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:41:13.959681 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:41:13.959681 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:41:13.959681 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:41:13.959681 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:41:13.959681 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:41:13.959681 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:41:13.959681 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:41:13.959681 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:41:13.959681 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:41:13.959681 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:41:13.959681 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:41:13.959681 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:41:13.959681 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:41:13.959681 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:41:13.959681 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:41:13.960457 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:41:13.960457 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:41:13.960457 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:41:13.960457 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:41:13.960457 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:41:13.960457 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:41:13.960457 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:41:13.960457 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:41:13.960457 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:41:13.960457 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:41:13.960457 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 25 00:41:13.960457 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:41:13.962202 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 78/155] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:41:13 2022] GET /identity/v3/roles/member => generated 84 bytes in 363 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:41:13.965592 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626dbce9-13eb-457d-923f-95084ed7fbf6 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:13.966764 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626dbce9-13eb-457d-923f-95084ed7fbf6 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:13.967357 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626dbce9-13eb-457d-923f-95084ed7fbf6 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:13.970256 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 79/156] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:13 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:13.972104 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-626dbce9-13eb-457d-923f-95084ed7fbf6 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:14.010322 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e943879a-72eb-48e2-8da6-be8e241959e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:14.011557 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e943879a-72eb-48e2-8da6-be8e241959e1 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:14.011994 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e943879a-72eb-48e2-8da6-be8e241959e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:14.012420 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e943879a-72eb-48e2-8da6-be8e241959e1 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:14.031832 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e943879a-72eb-48e2-8da6-be8e241959e1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:41:14.037558 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-e943879a-72eb-48e2-8da6-be8e241959e1 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:41:14.037558 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:41:14.037558 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:41:14.037558 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:41:14.037558 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:41:14.037558 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:41:14.037558 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:41:14.037558 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:41:14.037558 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:41:14.037558 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:41:14.037558 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:41:14.037558 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:41:14.037558 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:41:14.037558 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:41:14.037558 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:41:14.037558 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:41:14.038859 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:41:14.038859 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:41:14.038859 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:41:14.038859 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:41:14.038859 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:41:14.038859 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:41:14.038859 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:41:14.038859 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:41:14.038859 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:41:14.038859 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:41:14.038859 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:41:14.038859 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:41:14.038859 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:41:14.038859 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:41:14.038859 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:41:14.038859 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:41:14.039925 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:41:14.039925 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:41:14.039925 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:41:14.039925 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:41:14.039925 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:41:14.039925 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:41:14.039925 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:41:14.039925 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:41:14.039925 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:41:14.039925 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:41:14.039925 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:41:14.039925 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:41:14.043846 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 78/157] 10.222.0.52 () {56 vars in 1096 bytes} [Sun Sep 25 00:41:13 2022] GET /identity/v3/roles/admin => generated 83 bytes in 298 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:41:14.045518 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08db31e0-1062-4df5-86be-fd717f40af6a None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:14.045945 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08db31e0-1062-4df5-86be-fd717f40af6a None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:14.046379 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08db31e0-1062-4df5-86be-fd717f40af6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:14.077252 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-08db31e0-1062-4df5-86be-fd717f40af6a None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:14.080470 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-08db31e0-1062-4df5-86be-fd717f40af6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:14.166341 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-626dbce9-13eb-457d-923f-95084ed7fbf6 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:14.190774 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:14.221328 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f7ffb9-9458-4e8b-830c-d10b5d798d47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:14.223179 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f7ffb9-9458-4e8b-830c-d10b5d798d47 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:14.223791 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f7ffb9-9458-4e8b-830c-d10b5d798d47 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:14.224979 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f7ffb9-9458-4e8b-830c-d10b5d798d47 None admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:14.226365 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f7ffb9-9458-4e8b-830c-d10b5d798d47 None admin] RBAC: Adding query filter params (name=member) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:14.226971 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f7ffb9-9458-4e8b-830c-d10b5d798d47 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:14.230270 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f7ffb9-9458-4e8b-830c-d10b5d798d47 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:14.234942 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 79/158] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:41:14.235985 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08db31e0-1062-4df5-86be-fd717f40af6a None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:14.251605 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 80/159] 10.222.0.52 () {56 vars in 1101 bytes} [Sun Sep 25 00:41:13 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:14.252782 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f7ffb9-9458-4e8b-830c-d10b5d798d47 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:14.319619 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:14.392704 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08db31e0-1062-4df5-86be-fd717f40af6a None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:14.431190 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b60e1a-e330-49bc-8d37-22e864e84b55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:14.432265 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b60e1a-e330-49bc-8d37-22e864e84b55 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:14.432730 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b60e1a-e330-49bc-8d37-22e864e84b55 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:14.433185 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b60e1a-e330-49bc-8d37-22e864e84b55 None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:14.434085 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38b60e1a-e330-49bc-8d37-22e864e84b55 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:14.434586 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f7ffb9-9458-4e8b-830c-d10b5d798d47 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:14.434790 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38b60e1a-e330-49bc-8d37-22e864e84b55 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:14.436356 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38b60e1a-e330-49bc-8d37-22e864e84b55 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:14.446347 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 80/160] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:41:14 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:14.447612 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b60e1a-e330-49bc-8d37-22e864e84b55 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:14.499019 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:14.510544 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1885bbb3-f8c0-4767-bf2d-4a6b02979678 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:14.511204 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1885bbb3-f8c0-4767-bf2d-4a6b02979678 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:14.511488 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1885bbb3-f8c0-4767-bf2d-4a6b02979678 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:14.511758 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1885bbb3-f8c0-4767-bf2d-4a6b02979678 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:14.540555 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1885bbb3-f8c0-4767-bf2d-4a6b02979678 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:41:14.546248 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-1885bbb3-f8c0-4767-bf2d-4a6b02979678 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 25 00:41:14.546248 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:41:14.546248 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:41:14.546248 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:41:14.546248 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:41:14.546248 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:41:14.546248 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:41:14.546248 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:41:14.546248 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:41:14.546248 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:41:14.546248 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:41:14.546248 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:41:14.546248 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:41:14.546248 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:41:14.546248 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:41:14.546248 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:41:14.547158 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:41:14.547158 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:41:14.547158 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:41:14.547158 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:41:14.547158 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:41:14.547158 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:41:14.547158 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:41:14.547158 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:41:14.547158 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:41:14.547158 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:41:14.547158 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:41:14.547158 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:41:14.547158 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:41:14.547158 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:41:14.547158 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:41:14.547158 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:41:14.548417 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:41:14.548417 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:41:14.548417 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:41:14.548417 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:41:14.548417 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:41:14.548417 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:41:14.548417 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:41:14.548417 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:41:14.548417 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:41:14.548417 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:41:14.548417 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 25 00:41:14.548417 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:41:14.548417 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 81/161] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:41:14 2022] GET /identity/v3/roles/member => generated 84 bytes in 295 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:41:14.548417 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1885bbb3-f8c0-4767-bf2d-4a6b02979678 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:14.600129 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b60e1a-e330-49bc-8d37-22e864e84b55 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:14.665758 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e3ea816-9e76-4a91-a66e-11fc7189b761 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:14.666638 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3ea816-9e76-4a91-a66e-11fc7189b761 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:14.666907 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3ea816-9e76-4a91-a66e-11fc7189b761 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:14.667166 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3ea816-9e76-4a91-a66e-11fc7189b761 None admin] PATH_INFO: `/v3/users/6980cb57558a43d2b384c7b561d7af70` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:14.673996 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e3ea816-9e76-4a91-a66e-11fc7189b761 None admin] RBAC: Authorizing `identity:get_user(user_id=6980cb57558a43d2b384c7b561d7af70)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:14.676586 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e3ea816-9e76-4a91-a66e-11fc7189b761 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:14.678316 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 81/162] 10.222.0.52 () {56 vars in 1177 bytes} [Sun Sep 25 00:41:14 2022] GET /identity/v3/users/6980cb57558a43d2b384c7b561d7af70 => generated 275 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:14.679084 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e3ea816-9e76-4a91-a66e-11fc7189b761 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:14.745090 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:14.751135 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1885bbb3-f8c0-4767-bf2d-4a6b02979678 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:14.853029 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac519b2-c8aa-4c89-ba57-4203fe85e275 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:14.854233 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac519b2-c8aa-4c89-ba57-4203fe85e275 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:14.854708 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac519b2-c8aa-4c89-ba57-4203fe85e275 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:14.855126 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac519b2-c8aa-4c89-ba57-4203fe85e275 None admin] PATH_INFO: `/v3/users/d6d73e23626043529ac5b9e3889933d4` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:14.856283 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bac519b2-c8aa-4c89-ba57-4203fe85e275 None admin] RBAC: Authorizing `identity:get_user(user_id=d6d73e23626043529ac5b9e3889933d4)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:14.857768 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bac519b2-c8aa-4c89-ba57-4203fe85e275 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:14.859528 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 82/163] 10.222.0.52 () {56 vars in 1177 bytes} [Sun Sep 25 00:41:14 2022] GET /identity/v3/users/d6d73e23626043529ac5b9e3889933d4 => generated 247 bytes in 311 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:14.860092 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac519b2-c8aa-4c89-ba57-4203fe85e275 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:14.861778 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e3ea816-9e76-4a91-a66e-11fc7189b761 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:14.884904 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:14.950043 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf3a9f6-619f-4941-b5b2-41f4929f1b7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:14.950671 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf3a9f6-619f-4941-b5b2-41f4929f1b7c None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:14.950950 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf3a9f6-619f-4941-b5b2-41f4929f1b7c None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:14.951221 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf3a9f6-619f-4941-b5b2-41f4929f1b7c None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:14.951896 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbf3a9f6-619f-4941-b5b2-41f4929f1b7c None admin] RBAC: Adding query filter params (name=member) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:14.952195 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbf3a9f6-619f-4941-b5b2-41f4929f1b7c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:14.953620 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbf3a9f6-619f-4941-b5b2-41f4929f1b7c None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:14.971317 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 82/164] 10.222.0.52 () {56 vars in 1101 bytes} [Sun Sep 25 00:41:14 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 292 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:14.972178 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf3a9f6-619f-4941-b5b2-41f4929f1b7c None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:15.115588 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac519b2-c8aa-4c89-ba57-4203fe85e275 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:15.201804 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-437e2fe0-dac8-412c-8997-1b941207493d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:15.202412 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437e2fe0-dac8-412c-8997-1b941207493d None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:15.202686 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437e2fe0-dac8-412c-8997-1b941207493d None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:15.202954 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437e2fe0-dac8-412c-8997-1b941207493d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:15.215422 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf3a9f6-619f-4941-b5b2-41f4929f1b7c None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:15.272790 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc66041f-feac-46d4-85e9-01f397850d5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:15.273444 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc66041f-feac-46d4-85e9-01f397850d5c None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:15.273715 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc66041f-feac-46d4-85e9-01f397850d5c None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:15.274008 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc66041f-feac-46d4-85e9-01f397850d5c None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:15.291460 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc66041f-feac-46d4-85e9-01f397850d5c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:41:15.297220 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-dc66041f-feac-46d4-85e9-01f397850d5c None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 00:41:15.297220 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:41:15.297220 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:41:15.297220 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:41:15.297220 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:41:15.297220 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:41:15.297220 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:41:15.297220 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:41:15.297220 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:41:15.297220 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:41:15.297220 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:41:15.297220 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:41:15.297220 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:41:15.297220 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:41:15.297220 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:41:15.297220 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:41:15.298232 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:41:15.298232 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:41:15.298232 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:41:15.298232 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:41:15.298232 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:41:15.298232 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:41:15.298232 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:41:15.298232 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:41:15.298232 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:41:15.298232 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:41:15.298232 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:41:15.298232 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:41:15.298232 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:41:15.298232 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:41:15.298232 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:41:15.298232 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:41:15.299025 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:41:15.299025 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:41:15.299025 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:41:15.299025 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:41:15.299025 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:41:15.299025 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:41:15.299025 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:41:15.299025 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:41:15.299025 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:41:15.299025 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:41:15.299025 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 00:41:15.299025 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:41:15.299025 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 83/165] 10.222.0.52 () {56 vars in 1114 bytes} [Sun Sep 25 00:41:14 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 327 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:41:15.305605 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc66041f-feac-46d4-85e9-01f397850d5c None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:15.317637 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:15.359567 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:15.397666 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-437e2fe0-dac8-412c-8997-1b941207493d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:15.399207 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-437e2fe0-dac8-412c-8997-1b941207493d None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:15.532138 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc66041f-feac-46d4-85e9-01f397850d5c None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:15.589431 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 83/166] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:41:14 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 729 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 00:41:15.590169 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-437e2fe0-dac8-412c-8997-1b941207493d None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:15.614911 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ffe9398-386d-45fa-8169-07e539bcea5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:15.615901 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ffe9398-386d-45fa-8169-07e539bcea5a None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:15.616391 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ffe9398-386d-45fa-8169-07e539bcea5a None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:15.617054 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ffe9398-386d-45fa-8169-07e539bcea5a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:15.709946 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:15.782024 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-437e2fe0-dac8-412c-8997-1b941207493d None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:15.811721 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ffe9398-386d-45fa-8169-07e539bcea5a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:15.813611 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ffe9398-386d-45fa-8169-07e539bcea5a None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:15.871436 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa99da2c-20e1-4200-b06e-58893d194aa3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:15.872334 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa99da2c-20e1-4200-b06e-58893d194aa3 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:15.872750 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa99da2c-20e1-4200-b06e-58893d194aa3 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:15.873154 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa99da2c-20e1-4200-b06e-58893d194aa3 None admin] PATH_INFO: `/v3/users/6980cb57558a43d2b384c7b561d7af70` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:15.874521 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa99da2c-20e1-4200-b06e-58893d194aa3 None admin] RBAC: Authorizing `identity:get_user(user_id=6980cb57558a43d2b384c7b561d7af70)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:15.876127 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa99da2c-20e1-4200-b06e-58893d194aa3 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:15.877929 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 84/167] 10.222.0.52 () {56 vars in 1177 bytes} [Sun Sep 25 00:41:15 2022] GET /identity/v3/users/6980cb57558a43d2b384c7b561d7af70 => generated 275 bytes in 288 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:15.878818 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa99da2c-20e1-4200-b06e-58893d194aa3 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:15.888340 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:16.058011 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 84/168] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:41:15 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 759 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 00:41:16.059335 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ffe9398-386d-45fa-8169-07e539bcea5a None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:16.127040 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa99da2c-20e1-4200-b06e-58893d194aa3 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:16.196030 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd0385a8-c02b-4b0f-9109-b219e3b6fabe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:16.197705 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0385a8-c02b-4b0f-9109-b219e3b6fabe None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:16.198256 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0385a8-c02b-4b0f-9109-b219e3b6fabe None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:16.198735 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0385a8-c02b-4b0f-9109-b219e3b6fabe None admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:16.199629 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd0385a8-c02b-4b0f-9109-b219e3b6fabe None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:16.200131 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd0385a8-c02b-4b0f-9109-b219e3b6fabe None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:16.201765 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd0385a8-c02b-4b0f-9109-b219e3b6fabe None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:16.203631 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:16.220620 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 85/169] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:41:15 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 342 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:16.222405 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd0385a8-c02b-4b0f-9109-b219e3b6fabe None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:16.359062 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ffe9398-386d-45fa-8169-07e539bcea5a None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:16.430229 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:16.444061 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6794453e-c232-4082-bb78-0897e023f141 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:16.445572 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6794453e-c232-4082-bb78-0897e023f141 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:16.445572 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6794453e-c232-4082-bb78-0897e023f141 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:16.445572 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6794453e-c232-4082-bb78-0897e023f141 None admin] PATH_INFO: `/v3/projects/cd69c40698884b3192bb19ba6968842d` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:16.450893 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6794453e-c232-4082-bb78-0897e023f141 None admin] RBAC: Authorizing `identity:get_project(project_id=cd69c40698884b3192bb19ba6968842d)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:16.450893 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6794453e-c232-4082-bb78-0897e023f141 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:16.452719 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 85/170] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:41:16 2022] GET /identity/v3/projects/cd69c40698884b3192bb19ba6968842d => generated 298 bytes in 394 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:16.453459 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6794453e-c232-4082-bb78-0897e023f141 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:16.475635 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd0385a8-c02b-4b0f-9109-b219e3b6fabe None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:16.566460 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96a4450f-5019-4710-941b-1ec1bdb1f006 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:16.567438 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a4450f-5019-4710-941b-1ec1bdb1f006 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:16.567872 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a4450f-5019-4710-941b-1ec1bdb1f006 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:16.568313 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a4450f-5019-4710-941b-1ec1bdb1f006 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:16.701517 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:16.718699 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96a4450f-5019-4710-941b-1ec1bdb1f006 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:16.720399 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96a4450f-5019-4710-941b-1ec1bdb1f006 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:16.766042 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6794453e-c232-4082-bb78-0897e023f141 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:16.839540 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666ed663-c473-4685-81c2-502ad1c2c7e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:16.840179 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666ed663-c473-4685-81c2-502ad1c2c7e0 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:16.840455 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666ed663-c473-4685-81c2-502ad1c2c7e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:16.840722 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666ed663-c473-4685-81c2-502ad1c2c7e0 None admin] PATH_INFO: `/v3/projects/cd69c40698884b3192bb19ba6968842d` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:16.841931 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-666ed663-c473-4685-81c2-502ad1c2c7e0 None admin] RBAC: Authorizing `identity:get_project(project_id=cd69c40698884b3192bb19ba6968842d)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:16.843450 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-666ed663-c473-4685-81c2-502ad1c2c7e0 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:16.845420 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 86/171] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:41:16 2022] GET /identity/v3/projects/cd69c40698884b3192bb19ba6968842d => generated 298 bytes in 392 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:16.846731 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666ed663-c473-4685-81c2-502ad1c2c7e0 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:16.863376 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 86/172] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:41:16 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 642 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 00:41:16.870643 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96a4450f-5019-4710-941b-1ec1bdb1f006 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:16.932246 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:16.970472 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666ed663-c473-4685-81c2-502ad1c2c7e0 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:16.989991 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:16.999747 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96a4450f-5019-4710-941b-1ec1bdb1f006 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:17.015757 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22d8e497-fe14-4738-9bbf-80f65d9d6ffa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:17.016674 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d8e497-fe14-4738-9bbf-80f65d9d6ffa None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:17.017123 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d8e497-fe14-4738-9bbf-80f65d9d6ffa None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:17.017582 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d8e497-fe14-4738-9bbf-80f65d9d6ffa None admin] PATH_INFO: `/v3/users/6980cb57558a43d2b384c7b561d7af70` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:17.020163 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22d8e497-fe14-4738-9bbf-80f65d9d6ffa None admin] RBAC: Authorizing `identity:get_user(user_id=6980cb57558a43d2b384c7b561d7af70)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:17.022430 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22d8e497-fe14-4738-9bbf-80f65d9d6ffa None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:17.024538 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 87/173] 10.222.0.52 () {56 vars in 1177 bytes} [Sun Sep 25 00:41:16 2022] GET /identity/v3/users/6980cb57558a43d2b384c7b561d7af70 => generated 275 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:17.026121 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6992f267-5934-47fc-9072-eae75040a11a None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:17.026570 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6992f267-5934-47fc-9072-eae75040a11a None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:17.026981 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6992f267-5934-47fc-9072-eae75040a11a None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:17.028566 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 88/174] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:17.029844 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fceb63e-89ee-4769-8b6c-ea7c871a703b None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:17.030267 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fceb63e-89ee-4769-8b6c-ea7c871a703b None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:17.030698 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fceb63e-89ee-4769-8b6c-ea7c871a703b None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:17.032906 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 89/175] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:17.033935 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fceb63e-89ee-4769-8b6c-ea7c871a703b None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:17.057903 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193d360d-106f-4eb5-a390-b04bc2f29e7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:17.058529 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193d360d-106f-4eb5-a390-b04bc2f29e7f None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:17.058808 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193d360d-106f-4eb5-a390-b04bc2f29e7f None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:17.059072 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193d360d-106f-4eb5-a390-b04bc2f29e7f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:17.060364 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-193d360d-106f-4eb5-a390-b04bc2f29e7f None admin] RBAC: Adding query filter params (role.id=40f54f5ac0994c51960ac413da8d2cc9, scope.project.id=cd69c40698884b3192bb19ba6968842d, user.id=6980cb57558a43d2b384c7b561d7af70) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:17.060673 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-193d360d-106f-4eb5-a390-b04bc2f29e7f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:17.062429 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-193d360d-106f-4eb5-a390-b04bc2f29e7f None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:17.070574 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 87/176] 10.222.0.52 () {56 vars in 1374 bytes} [Sun Sep 25 00:41:16 2022] GET /identity/v3/role_assignments?user.id=6980cb57558a43d2b384c7b561d7af70&scope.project.id=cd69c40698884b3192bb19ba6968842d&role.id=40f54f5ac0994c51960ac413da8d2cc9 => generated 608 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:17.072830 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193d360d-106f-4eb5-a390-b04bc2f29e7f None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:17.190054 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fceb63e-89ee-4769-8b6c-ea7c871a703b None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:17.217632 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193d360d-106f-4eb5-a390-b04bc2f29e7f None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:17.235114 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7833431d-e471-4701-9361-f3bc8aac4f29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:17.235735 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7833431d-e471-4701-9361-f3bc8aac4f29 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:17.236014 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7833431d-e471-4701-9361-f3bc8aac4f29 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:17.236297 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7833431d-e471-4701-9361-f3bc8aac4f29 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:17.237029 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7833431d-e471-4701-9361-f3bc8aac4f29 None admin] RBAC: Adding query filter params (role.id=2c242abf6ed8410ab0a4a354ca1794fb, scope.project.id=cd69c40698884b3192bb19ba6968842d, user.id=d6d73e23626043529ac5b9e3889933d4) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:17.237369 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7833431d-e471-4701-9361-f3bc8aac4f29 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:17.239099 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7833431d-e471-4701-9361-f3bc8aac4f29 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:17.242874 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:17.253509 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 90/177] 10.222.0.52 () {56 vars in 1374 bytes} [Sun Sep 25 00:41:17 2022] GET /identity/v3/role_assignments?user.id=d6d73e23626043529ac5b9e3889933d4&scope.project.id=cd69c40698884b3192bb19ba6968842d&role.id=2c242abf6ed8410ab0a4a354ca1794fb => generated 608 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:17.253509 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a54c5fb-9085-4e15-8901-bb12a9cbbc4b None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:17.253509 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a54c5fb-9085-4e15-8901-bb12a9cbbc4b None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:17.253509 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a54c5fb-9085-4e15-8901-bb12a9cbbc4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:17.273866 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-1a54c5fb-9085-4e15-8901-bb12a9cbbc4b None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:17.274090 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a652f1f4-c849-4af1-a281-9ff546ffbade None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:17.274568 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a652f1f4-c849-4af1-a281-9ff546ffbade None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:17.274849 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a652f1f4-c849-4af1-a281-9ff546ffbade None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:17.275669 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a652f1f4-c849-4af1-a281-9ff546ffbade None admin] PATH_INFO: `/v3/projects/8ad3c16a01344fc6a722a172049e9b0f` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:17.276601 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-1a54c5fb-9085-4e15-8901-bb12a9cbbc4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:17.276908 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a652f1f4-c849-4af1-a281-9ff546ffbade None admin] RBAC: Authorizing `identity:get_project(project_id=8ad3c16a01344fc6a722a172049e9b0f)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:17.278482 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a652f1f4-c849-4af1-a281-9ff546ffbade None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:17.282275 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 88/178] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:41:17 2022] GET /identity/v3/projects/8ad3c16a01344fc6a722a172049e9b0f => generated 312 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:17.282495 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a652f1f4-c849-4af1-a281-9ff546ffbade None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:17.332931 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:17.376062 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 91/179] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:41:17.378344 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1004541e-af51-4bec-b2ea-90566d351ca7 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:17.378641 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1004541e-af51-4bec-b2ea-90566d351ca7 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:17.378909 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1004541e-af51-4bec-b2ea-90566d351ca7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:17.398550 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-1004541e-af51-4bec-b2ea-90566d351ca7 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:17.399776 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-1004541e-af51-4bec-b2ea-90566d351ca7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:17.426617 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a652f1f4-c849-4af1-a281-9ff546ffbade None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:17.474214 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18198fd1-49ba-4615-b74d-205e7b76a7cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:17.474544 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18198fd1-49ba-4615-b74d-205e7b76a7cb None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:17.474728 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18198fd1-49ba-4615-b74d-205e7b76a7cb None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:17.475163 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18198fd1-49ba-4615-b74d-205e7b76a7cb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:17.495334 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 92/180] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:41:17.498580 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fea6335-aea0-4aba-8fc1-2e06ad18b4c8 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:17.498580 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fea6335-aea0-4aba-8fc1-2e06ad18b4c8 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:17.498580 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fea6335-aea0-4aba-8fc1-2e06ad18b4c8 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:17.499265 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 93/181] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:17.504626 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866a4723-2af0-44d9-a793-df66aa3bf168 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:17.504868 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866a4723-2af0-44d9-a793-df66aa3bf168 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:17.505112 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866a4723-2af0-44d9-a793-df66aa3bf168 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:17.506578 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 94/182] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:17 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:17.507129 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866a4723-2af0-44d9-a793-df66aa3bf168 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:17.576570 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18198fd1-49ba-4615-b74d-205e7b76a7cb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:17.578449 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18198fd1-49ba-4615-b74d-205e7b76a7cb None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:17.594834 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:17.620341 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:17.642550 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866a4723-2af0-44d9-a793-df66aa3bf168 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:17.687620 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72590eca-05bd-46f3-a185-e87420c8656d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:17.688907 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72590eca-05bd-46f3-a185-e87420c8656d None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:17.689385 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72590eca-05bd-46f3-a185-e87420c8656d None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:17.690122 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72590eca-05bd-46f3-a185-e87420c8656d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:17.691334 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72590eca-05bd-46f3-a185-e87420c8656d None admin] RBAC: Adding query filter params (role.id=40f54f5ac0994c51960ac413da8d2cc9, scope.project.id=8ad3c16a01344fc6a722a172049e9b0f, user.id=6980cb57558a43d2b384c7b561d7af70) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:17.691817 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72590eca-05bd-46f3-a185-e87420c8656d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:17.694147 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72590eca-05bd-46f3-a185-e87420c8656d None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:17.702625 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 95/183] 10.222.0.52 () {56 vars in 1374 bytes} [Sun Sep 25 00:41:17 2022] GET /identity/v3/role_assignments?user.id=6980cb57558a43d2b384c7b561d7af70&scope.project.id=8ad3c16a01344fc6a722a172049e9b0f&role.id=40f54f5ac0994c51960ac413da8d2cc9 => generated 608 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:17.704107 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac78e89b-be7f-4d18-b55e-885ebd924c7c None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:17.704617 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac78e89b-be7f-4d18-b55e-885ebd924c7c None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:17.705060 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac78e89b-be7f-4d18-b55e-885ebd924c7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:17.723401 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 89/184] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:41:17 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 441 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 00:41:17.724426 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717b5c5f-6764-4d22-bed6-b41b609ff85d None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:17.724791 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717b5c5f-6764-4d22-bed6-b41b609ff85d None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:17.725126 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717b5c5f-6764-4d22-bed6-b41b609ff85d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:17.730815 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-ac78e89b-be7f-4d18-b55e-885ebd924c7c None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:17.732338 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-ac78e89b-be7f-4d18-b55e-885ebd924c7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:17.747279 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-717b5c5f-6764-4d22-bed6-b41b609ff85d None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:17.748659 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-717b5c5f-6764-4d22-bed6-b41b609ff85d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:17.822796 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 96/185] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:41:17.824356 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac78e89b-be7f-4d18-b55e-885ebd924c7c None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:17.836373 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 90/186] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:41:17.840807 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-717b5c5f-6764-4d22-bed6-b41b609ff85d None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:17.905869 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:17.920810 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:17.965729 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac78e89b-be7f-4d18-b55e-885ebd924c7c None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:17.985367 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-717b5c5f-6764-4d22-bed6-b41b609ff85d None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:18.014644 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5a3f2b-d6f4-4bf0-b9a8-4982476644f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:18.015420 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5a3f2b-d6f4-4bf0-b9a8-4982476644f8 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:18.015701 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5a3f2b-d6f4-4bf0-b9a8-4982476644f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:18.015974 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5a3f2b-d6f4-4bf0-b9a8-4982476644f8 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:18.031638 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae5a3f2b-d6f4-4bf0-b9a8-4982476644f8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:41:18.032442 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c5d65f-f82a-4d1c-80ef-a4ba2537a52a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:18.033237 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c5d65f-f82a-4d1c-80ef-a4ba2537a52a None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:18.033732 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c5d65f-f82a-4d1c-80ef-a4ba2537a52a None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:18.034146 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c5d65f-f82a-4d1c-80ef-a4ba2537a52a None admin] PATH_INFO: `/v3/projects/cd69c40698884b3192bb19ba6968842d` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:18.035434 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6c5d65f-f82a-4d1c-80ef-a4ba2537a52a None admin] RBAC: Authorizing `identity:get_project(project_id=cd69c40698884b3192bb19ba6968842d)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:18.036729 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-ae5a3f2b-d6f4-4bf0-b9a8-4982476644f8 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:41:18.036729 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:41:18.036729 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:41:18.036729 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:41:18.036729 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:41:18.036729 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:41:18.036729 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:41:18.036729 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:41:18.036729 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:41:18.036729 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:41:18.036729 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:41:18.036729 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:41:18.036729 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:41:18.036729 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:41:18.036729 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:41:18.036729 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:41:18.037839 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:41:18.037839 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:41:18.037839 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:41:18.037839 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:41:18.037839 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:41:18.037839 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:41:18.037839 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:41:18.037839 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:41:18.037839 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:41:18.037839 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:41:18.037839 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:41:18.037839 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:41:18.037839 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:41:18.037839 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:41:18.037839 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:41:18.037839 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:41:18.038650 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:41:18.038650 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:41:18.038650 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:41:18.038650 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:41:18.038650 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:41:18.038650 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:41:18.038650 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:41:18.038650 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:41:18.038650 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:41:18.038650 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:41:18.038650 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:41:18.038650 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:41:18.038650 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 97/187] 10.222.0.52 () {56 vars in 1096 bytes} [Sun Sep 25 00:41:17 2022] GET /identity/v3/roles/admin => generated 83 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:41:18.039390 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5a3f2b-d6f4-4bf0-b9a8-4982476644f8 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:18.040215 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6c5d65f-f82a-4d1c-80ef-a4ba2537a52a None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:18.042517 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 91/188] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:41:17 2022] GET /identity/v3/projects/cd69c40698884b3192bb19ba6968842d => generated 298 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:18.044475 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c5d65f-f82a-4d1c-80ef-a4ba2537a52a None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:18.172784 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c5d65f-f82a-4d1c-80ef-a4ba2537a52a None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:18.177687 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5a3f2b-d6f4-4bf0-b9a8-4982476644f8 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:18.193700 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:18.201498 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:18.210982 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9990bf25-56b8-44d9-a243-5757d67417b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:18.211577 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9990bf25-56b8-44d9-a243-5757d67417b6 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:18.211846 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9990bf25-56b8-44d9-a243-5757d67417b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:18.212119 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9990bf25-56b8-44d9-a243-5757d67417b6 None admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:18.213522 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9990bf25-56b8-44d9-a243-5757d67417b6 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:18.214052 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9990bf25-56b8-44d9-a243-5757d67417b6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:18.215482 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9990bf25-56b8-44d9-a243-5757d67417b6 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:18.217628 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ad1f44e-79a4-4166-a5d5-877459e1c068 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:18.218253 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ad1f44e-79a4-4166-a5d5-877459e1c068 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:18.218526 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ad1f44e-79a4-4166-a5d5-877459e1c068 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:18.218792 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ad1f44e-79a4-4166-a5d5-877459e1c068 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:18.223778 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 92/189] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:41:18 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:18.224396 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9990bf25-56b8-44d9-a243-5757d67417b6 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:18.232556 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ad1f44e-79a4-4166-a5d5-877459e1c068 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:41:18.237517 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-6ad1f44e-79a4-4166-a5d5-877459e1c068 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 25 00:41:18.237517 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:41:18.237517 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:41:18.237517 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:41:18.237517 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:41:18.237517 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:41:18.237517 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:41:18.237517 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:41:18.237517 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:41:18.237517 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:41:18.237517 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:41:18.237517 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:41:18.237517 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:41:18.237517 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:41:18.237517 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:41:18.237517 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:41:18.238778 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:41:18.238778 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:41:18.238778 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:41:18.238778 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:41:18.238778 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:41:18.238778 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:41:18.238778 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:41:18.238778 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:41:18.238778 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:41:18.238778 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:41:18.238778 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:41:18.238778 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:41:18.238778 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:41:18.238778 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:41:18.238778 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:41:18.238778 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:41:18.239578 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:41:18.239578 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:41:18.239578 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:41:18.239578 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:41:18.239578 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:41:18.239578 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:41:18.239578 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:41:18.239578 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:41:18.239578 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:41:18.239578 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:41:18.239578 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 25 00:41:18.239578 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:41:18.239578 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 98/190] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:41:18 2022] GET /identity/v3/roles/reader => generated 84 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:41:18.240266 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ad1f44e-79a4-4166-a5d5-877459e1c068 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:18.333050 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9990bf25-56b8-44d9-a243-5757d67417b6 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:18.343884 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ad1f44e-79a4-4166-a5d5-877459e1c068 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:18.366450 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1faf6c20-8a2e-47d7-8fea-f9a87cd3e5f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:18.367054 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1faf6c20-8a2e-47d7-8fea-f9a87cd3e5f4 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:18.367331 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1faf6c20-8a2e-47d7-8fea-f9a87cd3e5f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:18.367598 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1faf6c20-8a2e-47d7-8fea-f9a87cd3e5f4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:18.368310 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1faf6c20-8a2e-47d7-8fea-f9a87cd3e5f4 None admin] RBAC: Adding query filter params (role.id=a917909d7aa84d00a841c77e7e7bdc8a, scope.project.id=cd69c40698884b3192bb19ba6968842d, user.id=6980cb57558a43d2b384c7b561d7af70) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:18.368615 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1faf6c20-8a2e-47d7-8fea-f9a87cd3e5f4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:18.370594 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1faf6c20-8a2e-47d7-8fea-f9a87cd3e5f4 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:18.376399 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 93/191] 10.222.0.52 () {56 vars in 1374 bytes} [Sun Sep 25 00:41:18 2022] GET /identity/v3/role_assignments?user.id=6980cb57558a43d2b384c7b561d7af70&scope.project.id=cd69c40698884b3192bb19ba6968842d&role.id=a917909d7aa84d00a841c77e7e7bdc8a => generated 608 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:18.377176 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1faf6c20-8a2e-47d7-8fea-f9a87cd3e5f4 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:18.378209 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bda20359-2adf-449b-affd-5b3c3d70021b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:18.378834 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda20359-2adf-449b-affd-5b3c3d70021b None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:18.379105 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda20359-2adf-449b-affd-5b3c3d70021b None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:18.379395 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda20359-2adf-449b-affd-5b3c3d70021b None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:18.392000 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bda20359-2adf-449b-affd-5b3c3d70021b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:41:18.397349 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-bda20359-2adf-449b-affd-5b3c3d70021b None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 00:41:18.397349 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:41:18.397349 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:41:18.397349 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:41:18.397349 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:41:18.397349 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:41:18.397349 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:41:18.397349 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:41:18.397349 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:41:18.397349 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:41:18.397349 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:41:18.397349 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:41:18.397349 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:41:18.397349 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:41:18.397349 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:41:18.397349 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:41:18.398371 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:41:18.398371 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:41:18.398371 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:41:18.398371 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:41:18.398371 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:41:18.398371 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:41:18.398371 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:41:18.398371 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:41:18.398371 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:41:18.398371 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:41:18.398371 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:41:18.398371 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:41:18.398371 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:41:18.398371 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:41:18.398371 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:41:18.398371 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:41:18.401732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:41:18.401732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:41:18.401732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:41:18.401732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:41:18.401732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:41:18.401732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:41:18.401732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:41:18.401732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:41:18.401732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:41:18.401732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:41:18.401732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 00:41:18.401732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:41:18.401732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 99/192] 10.222.0.52 () {56 vars in 1114 bytes} [Sun Sep 25 00:41:18 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:41:18.401732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bda20359-2adf-449b-affd-5b3c3d70021b None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:18.443993 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:18.445816 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:18.514307 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1faf6c20-8a2e-47d7-8fea-f9a87cd3e5f4 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:18.524680 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bda20359-2adf-449b-affd-5b3c3d70021b None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:18.552562 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9436e370-f369-4e9f-9690-94c05e98bc4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:18.553207 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9436e370-f369-4e9f-9690-94c05e98bc4f None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:18.553514 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9436e370-f369-4e9f-9690-94c05e98bc4f None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:18.553782 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9436e370-f369-4e9f-9690-94c05e98bc4f None admin] PATH_INFO: `/v3/domains/default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:18.554924 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9436e370-f369-4e9f-9690-94c05e98bc4f None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:18.556978 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9436e370-f369-4e9f-9690-94c05e98bc4f None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:18.563247 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 94/193] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:41:18 2022] GET /identity/v3/domains/default => generated 199 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:18.563247 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9436e370-f369-4e9f-9690-94c05e98bc4f None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:18.580396 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9589889-7fdc-4056-99e9-c468234cd621 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:18.581117 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9589889-7fdc-4056-99e9-c468234cd621 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:18.581338 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9589889-7fdc-4056-99e9-c468234cd621 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:18.581607 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9589889-7fdc-4056-99e9-c468234cd621 None admin] PATH_INFO: `/v3/users/8ed437642b0742d4a84d4f8432538145` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:18.596351 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9589889-7fdc-4056-99e9-c468234cd621 None admin] RBAC: Authorizing `identity:get_user(user_id=8ed437642b0742d4a84d4f8432538145)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:18.598104 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9589889-7fdc-4056-99e9-c468234cd621 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:18.600891 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 100/194] 10.222.0.52 () {56 vars in 1177 bytes} [Sun Sep 25 00:41:18 2022] GET /identity/v3/users/8ed437642b0742d4a84d4f8432538145 => generated 283 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:18.606175 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9589889-7fdc-4056-99e9-c468234cd621 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:18.673919 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9436e370-f369-4e9f-9690-94c05e98bc4f None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:18.693340 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:18.706106 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:18.708971 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af49717-251e-40c2-b521-ba1e77351a17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:18.709608 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af49717-251e-40c2-b521-ba1e77351a17 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:18.709886 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af49717-251e-40c2-b521-ba1e77351a17 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:18.710154 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af49717-251e-40c2-b521-ba1e77351a17 None admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:18.710816 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1af49717-251e-40c2-b521-ba1e77351a17 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:18.711117 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1af49717-251e-40c2-b521-ba1e77351a17 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:18.712489 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1af49717-251e-40c2-b521-ba1e77351a17 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:18.716777 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9589889-7fdc-4056-99e9-c468234cd621 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:18.721686 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 95/195] 10.222.0.52 () {56 vars in 1101 bytes} [Sun Sep 25 00:41:18 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:18.722335 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af49717-251e-40c2-b521-ba1e77351a17 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:18.753236 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd28165-3c62-4a66-988e-1bbfc109cc0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:18.753873 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd28165-3c62-4a66-988e-1bbfc109cc0a None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:18.754146 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd28165-3c62-4a66-988e-1bbfc109cc0a None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:18.754414 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd28165-3c62-4a66-988e-1bbfc109cc0a None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:18.755072 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fd28165-3c62-4a66-988e-1bbfc109cc0a None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:18.755433 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fd28165-3c62-4a66-988e-1bbfc109cc0a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:18.756801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fd28165-3c62-4a66-988e-1bbfc109cc0a None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:18.766362 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 101/196] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:41:18 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:18.767165 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd28165-3c62-4a66-988e-1bbfc109cc0a None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:18.829432 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af49717-251e-40c2-b521-ba1e77351a17 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:18.863291 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be7870f-0078-49e8-b4a4-e73d79b42e3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:18.863821 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be7870f-0078-49e8-b4a4-e73d79b42e3e None admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:18.864218 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be7870f-0078-49e8-b4a4-e73d79b42e3e None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:18.864494 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be7870f-0078-49e8-b4a4-e73d79b42e3e None admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:18.865143 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5be7870f-0078-49e8-b4a4-e73d79b42e3e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:18.865538 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd28165-3c62-4a66-988e-1bbfc109cc0a None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:18.866660 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5be7870f-0078-49e8-b4a4-e73d79b42e3e None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:18.888496 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 96/197] 10.222.0.52 () {60 vars in 1132 bytes} [Sun Sep 25 00:41:18 2022] POST /identity/v3/users => generated 297 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:41:18.889337 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be7870f-0078-49e8-b4a4-e73d79b42e3e None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:18.905222 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bfd85aa-68ce-482c-8194-37a5a41ccd57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:18.905884 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfd85aa-68ce-482c-8194-37a5a41ccd57 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:18.906163 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfd85aa-68ce-482c-8194-37a5a41ccd57 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:18.906435 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfd85aa-68ce-482c-8194-37a5a41ccd57 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:18.934973 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:18.938382 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:18.993534 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bfd85aa-68ce-482c-8194-37a5a41ccd57 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:18.995342 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bfd85aa-68ce-482c-8194-37a5a41ccd57 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:19.011408 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be7870f-0078-49e8-b4a4-e73d79b42e3e None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:19.048611 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dc79776-0c63-4354-b7dd-0d0aede34456 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:19.049266 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc79776-0c63-4354-b7dd-0d0aede34456 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:19.049551 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc79776-0c63-4354-b7dd-0d0aede34456 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:19.049823 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc79776-0c63-4354-b7dd-0d0aede34456 None admin] PATH_INFO: `/v3/users/f079800ac9f04bf2b626bb34da78bd52` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:19.051017 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dc79776-0c63-4354-b7dd-0d0aede34456 None admin] RBAC: Authorizing `identity:get_user(user_id=f079800ac9f04bf2b626bb34da78bd52)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:19.052497 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dc79776-0c63-4354-b7dd-0d0aede34456 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:19.054344 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 97/198] 10.222.0.52 () {56 vars in 1177 bytes} [Sun Sep 25 00:41:18 2022] GET /identity/v3/users/f079800ac9f04bf2b626bb34da78bd52 => generated 289 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:19.054960 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dc79776-0c63-4354-b7dd-0d0aede34456 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:19.113652 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 102/199] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:41:18 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 00:41:19.114420 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bfd85aa-68ce-482c-8194-37a5a41ccd57 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:19.169643 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dc79776-0c63-4354-b7dd-0d0aede34456 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:19.178664 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:19.190658 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:19.220320 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51879dab-6e98-4fe3-89ac-0a6b20ceceb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:19.220320 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51879dab-6e98-4fe3-89ac-0a6b20ceceb6 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:19.220320 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51879dab-6e98-4fe3-89ac-0a6b20ceceb6 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:19.220320 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51879dab-6e98-4fe3-89ac-0a6b20ceceb6 None admin] PATH_INFO: `/v3/users/8ed437642b0742d4a84d4f8432538145` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:19.221980 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51879dab-6e98-4fe3-89ac-0a6b20ceceb6 None admin] RBAC: Authorizing `identity:get_user(user_id=8ed437642b0742d4a84d4f8432538145)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:19.223567 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51879dab-6e98-4fe3-89ac-0a6b20ceceb6 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:19.225361 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 98/200] 10.222.0.52 () {56 vars in 1177 bytes} [Sun Sep 25 00:41:19 2022] GET /identity/v3/users/8ed437642b0742d4a84d4f8432538145 => generated 283 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:19.226145 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51879dab-6e98-4fe3-89ac-0a6b20ceceb6 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:19.252996 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bfd85aa-68ce-482c-8194-37a5a41ccd57 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:19.309043 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa00358-a692-4d5d-8866-305477f0aa65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:19.309703 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa00358-a692-4d5d-8866-305477f0aa65 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:19.309981 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa00358-a692-4d5d-8866-305477f0aa65 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:19.310252 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa00358-a692-4d5d-8866-305477f0aa65 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:19.342557 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51879dab-6e98-4fe3-89ac-0a6b20ceceb6 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:19.391505 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa00358-a692-4d5d-8866-305477f0aa65 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:19.391804 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f15b893-852f-40d9-9a80-37143e074891 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:19.392379 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f15b893-852f-40d9-9a80-37143e074891 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:19.392665 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f15b893-852f-40d9-9a80-37143e074891 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:19.392939 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f15b893-852f-40d9-9a80-37143e074891 None admin] PATH_INFO: `/v3/projects/86aea44da5fb4bdeb527c7ff257c5d43` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:19.393040 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa00358-a692-4d5d-8866-305477f0aa65 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:19.394114 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f15b893-852f-40d9-9a80-37143e074891 None admin] RBAC: Authorizing `identity:get_project(project_id=86aea44da5fb4bdeb527c7ff257c5d43)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:19.396739 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f15b893-852f-40d9-9a80-37143e074891 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:19.398773 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 99/201] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:41:19 2022] GET /identity/v3/projects/86aea44da5fb4bdeb527c7ff257c5d43 => generated 302 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:19.399558 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f15b893-852f-40d9-9a80-37143e074891 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:19.454147 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:19.463627 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:19.507834 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 103/202] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:41:19 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 394 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 00:41:19.513415 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa00358-a692-4d5d-8866-305477f0aa65 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:19.545911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f15b893-852f-40d9-9a80-37143e074891 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:19.586026 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5099d702-fff2-461f-b26c-f1e6c692a2e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:19.587643 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5099d702-fff2-461f-b26c-f1e6c692a2e8 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:19.588196 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5099d702-fff2-461f-b26c-f1e6c692a2e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:19.588643 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5099d702-fff2-461f-b26c-f1e6c692a2e8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:19.637402 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa00358-a692-4d5d-8866-305477f0aa65 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:19.679106 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5099d702-fff2-461f-b26c-f1e6c692a2e8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:19.679851 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d9e98e1-2471-4ae4-83c5-1368d5c3d5f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:19.680576 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9e98e1-2471-4ae4-83c5-1368d5c3d5f3 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:19.680948 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9e98e1-2471-4ae4-83c5-1368d5c3d5f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:19.681384 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9e98e1-2471-4ae4-83c5-1368d5c3d5f3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:19.681626 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5099d702-fff2-461f-b26c-f1e6c692a2e8 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:19.682415 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d9e98e1-2471-4ae4-83c5-1368d5c3d5f3 None admin] RBAC: Adding query filter params (role.id=2c242abf6ed8410ab0a4a354ca1794fb, scope.project.id=86aea44da5fb4bdeb527c7ff257c5d43, user.id=8ed437642b0742d4a84d4f8432538145) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:19.682815 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d9e98e1-2471-4ae4-83c5-1368d5c3d5f3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:19.684462 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d9e98e1-2471-4ae4-83c5-1368d5c3d5f3 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:19.690721 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 104/203] 10.222.0.52 () {56 vars in 1374 bytes} [Sun Sep 25 00:41:19 2022] GET /identity/v3/role_assignments?user.id=8ed437642b0742d4a84d4f8432538145&scope.project.id=86aea44da5fb4bdeb527c7ff257c5d43&role.id=2c242abf6ed8410ab0a4a354ca1794fb => generated 259 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:19.691629 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d9e98e1-2471-4ae4-83c5-1368d5c3d5f3 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:19.716391 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:19.745644 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:19.816651 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d9e98e1-2471-4ae4-83c5-1368d5c3d5f3 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:19.834925 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 100/204] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:41:19 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 436 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 00:41:19.844647 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5099d702-fff2-461f-b26c-f1e6c692a2e8 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:19.862744 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cef8926-c67f-4bb1-af5e-efe03da944f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:19.864543 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cef8926-c67f-4bb1-af5e-efe03da944f0 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:19.865053 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cef8926-c67f-4bb1-af5e-efe03da944f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:19.865585 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cef8926-c67f-4bb1-af5e-efe03da944f0 None admin] PATH_INFO: `/v3/projects/cd69c40698884b3192bb19ba6968842d` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:19.867657 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cef8926-c67f-4bb1-af5e-efe03da944f0 None admin] RBAC: Authorizing `identity:get_project(project_id=cd69c40698884b3192bb19ba6968842d)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:19.869475 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cef8926-c67f-4bb1-af5e-efe03da944f0 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:19.872643 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 105/205] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:41:19 2022] GET /identity/v3/projects/cd69c40698884b3192bb19ba6968842d => generated 298 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:19.879089 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cef8926-c67f-4bb1-af5e-efe03da944f0 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:19.994172 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:20.014800 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cef8926-c67f-4bb1-af5e-efe03da944f0 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:20.057643 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4c808d-8517-4d5a-aa6b-9fb4c6d3ebe3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:20.058645 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4c808d-8517-4d5a-aa6b-9fb4c6d3ebe3 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:20.059127 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4c808d-8517-4d5a-aa6b-9fb4c6d3ebe3 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:20.059927 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4c808d-8517-4d5a-aa6b-9fb4c6d3ebe3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:20.060866 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac4c808d-8517-4d5a-aa6b-9fb4c6d3ebe3 None admin] RBAC: Adding query filter params (role.id=cc65bfe8ed2740b69427f705add475a5, scope.project.id=cd69c40698884b3192bb19ba6968842d, user.id=f079800ac9f04bf2b626bb34da78bd52) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:20.061677 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac4c808d-8517-4d5a-aa6b-9fb4c6d3ebe3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:20.064153 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac4c808d-8517-4d5a-aa6b-9fb4c6d3ebe3 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:20.073287 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 106/206] 10.222.0.52 () {56 vars in 1374 bytes} [Sun Sep 25 00:41:19 2022] GET /identity/v3/role_assignments?user.id=f079800ac9f04bf2b626bb34da78bd52&scope.project.id=cd69c40698884b3192bb19ba6968842d&role.id=cc65bfe8ed2740b69427f705add475a5 => generated 608 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:41:20.090245 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5099d702-fff2-461f-b26c-f1e6c692a2e8 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:20.143262 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ede3c1-7503-4b94-a253-b253dc2965e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:20.143918 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ede3c1-7503-4b94-a253-b253dc2965e7 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:20.144193 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ede3c1-7503-4b94-a253-b253dc2965e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:20.144461 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ede3c1-7503-4b94-a253-b253dc2965e7 None admin] PATH_INFO: `/v3/projects/86aea44da5fb4bdeb527c7ff257c5d43` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:20.145684 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20ede3c1-7503-4b94-a253-b253dc2965e7 None admin] RBAC: Authorizing `identity:get_project(project_id=86aea44da5fb4bdeb527c7ff257c5d43)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:20.147186 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20ede3c1-7503-4b94-a253-b253dc2965e7 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:20.149203 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 101/207] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:41:19 2022] GET /identity/v3/projects/86aea44da5fb4bdeb527c7ff257c5d43 => generated 302 bytes in 305 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:20.158616 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4c808d-8517-4d5a-aa6b-9fb4c6d3ebe3 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:20.281516 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4c808d-8517-4d5a-aa6b-9fb4c6d3ebe3 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:20.320560 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c275dfad-47b3-4866-956c-d6087db5b6a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:20.321462 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c275dfad-47b3-4866-956c-d6087db5b6a8 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:20.321941 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c275dfad-47b3-4866-956c-d6087db5b6a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:20.322881 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c275dfad-47b3-4866-956c-d6087db5b6a8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:20.323783 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c275dfad-47b3-4866-956c-d6087db5b6a8 None admin] RBAC: Adding query filter params (role.id=a917909d7aa84d00a841c77e7e7bdc8a, scope.project.id=86aea44da5fb4bdeb527c7ff257c5d43, user.id=8ed437642b0742d4a84d4f8432538145) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:20.324238 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c275dfad-47b3-4866-956c-d6087db5b6a8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:20.325870 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c275dfad-47b3-4866-956c-d6087db5b6a8 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:20.332760 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 107/208] 10.222.0.52 () {56 vars in 1374 bytes} [Sun Sep 25 00:41:20 2022] GET /identity/v3/role_assignments?user.id=8ed437642b0742d4a84d4f8432538145&scope.project.id=86aea44da5fb4bdeb527c7ff257c5d43&role.id=a917909d7aa84d00a841c77e7e7bdc8a => generated 259 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:21.382410 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b531a22-62a1-45e2-b3ce-942677d359a6 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:21.382932 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b531a22-62a1-45e2-b3ce-942677d359a6 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:21.382932 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b531a22-62a1-45e2-b3ce-942677d359a6 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:21.384389 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 102/209] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:21.390704 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4fc20a-5f8b-4d38-b6d6-c50d935f1a17 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:21.391157 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4fc20a-5f8b-4d38-b6d6-c50d935f1a17 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:21.391261 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4fc20a-5f8b-4d38-b6d6-c50d935f1a17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:21.411765 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-0f4fc20a-5f8b-4d38-b6d6-c50d935f1a17 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:21.413024 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-0f4fc20a-5f8b-4d38-b6d6-c50d935f1a17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:21.426088 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:21.495034 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 108/210] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:41:21.539329 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66605686-54ef-42f2-82cd-5f0f7d347703 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:21.539604 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66605686-54ef-42f2-82cd-5f0f7d347703 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:21.539874 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66605686-54ef-42f2-82cd-5f0f7d347703 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:21.542112 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 103/211] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:21.547560 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae19ab2e-dd23-4e1b-9190-29ce0918a514 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:21.547848 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae19ab2e-dd23-4e1b-9190-29ce0918a514 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:21.548115 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae19ab2e-dd23-4e1b-9190-29ce0918a514 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:21.568052 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-ae19ab2e-dd23-4e1b-9190-29ce0918a514 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:21.569245 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-ae19ab2e-dd23-4e1b-9190-29ce0918a514 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:21.646004 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 109/212] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:41:21.770377 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66605686-54ef-42f2-82cd-5f0f7d347703 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:21.784021 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:21.880926 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66605686-54ef-42f2-82cd-5f0f7d347703 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:21.921300 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-728dde92-13d5-4919-81f4-d1d2402b129d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:21.921851 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-728dde92-13d5-4919-81f4-d1d2402b129d None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:21.922124 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-728dde92-13d5-4919-81f4-d1d2402b129d None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:21.922390 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-728dde92-13d5-4919-81f4-d1d2402b129d None admin] PATH_INFO: `/v3/roles/member` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:21.933492 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-728dde92-13d5-4919-81f4-d1d2402b129d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:41:21.938337 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-728dde92-13d5-4919-81f4-d1d2402b129d None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 25 00:41:21.938337 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:41:21.938337 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:41:21.938337 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:41:21.938337 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:41:21.938337 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:41:21.938337 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:41:21.938337 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:41:21.938337 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:41:21.938337 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:41:21.938337 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:41:21.938337 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:41:21.938337 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:41:21.938337 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:41:21.938337 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:41:21.938337 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:41:21.939258 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:41:21.939258 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:41:21.939258 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:41:21.939258 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:41:21.939258 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:41:21.939258 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:41:21.939258 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:41:21.939258 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:41:21.939258 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:41:21.939258 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:41:21.939258 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:41:21.939258 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:41:21.939258 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:41:21.939258 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:41:21.939258 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:41:21.939258 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:41:21.940485 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:41:21.940485 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:41:21.940485 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:41:21.940485 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:41:21.940485 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:41:21.940485 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:41:21.940485 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:41:21.940485 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:41:21.940485 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:41:21.940485 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:41:21.940485 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 25 00:41:21.940485 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:41:21.941968 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae19ab2e-dd23-4e1b-9190-29ce0918a514 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:21.942112 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 104/213] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:41:21 2022] GET /identity/v3/roles/member => generated 84 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:41:21.945274 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-728dde92-13d5-4919-81f4-d1d2402b129d None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:22.020136 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:22.033144 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:22.056066 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-728dde92-13d5-4919-81f4-d1d2402b129d None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:22.064715 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae19ab2e-dd23-4e1b-9190-29ce0918a514 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:22.109535 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc71b16-45f7-406e-b19f-946280d67b04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:22.110230 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc71b16-45f7-406e-b19f-946280d67b04 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:22.110508 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc71b16-45f7-406e-b19f-946280d67b04 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:22.110776 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc71b16-45f7-406e-b19f-946280d67b04 None admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:22.111446 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc71b16-45f7-406e-b19f-946280d67b04 None admin] RBAC: Adding query filter params (name=member) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:22.111733 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76aa3874-2668-40b4-93cf-4a29028a98e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:22.111868 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc71b16-45f7-406e-b19f-946280d67b04 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:22.112543 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76aa3874-2668-40b4-93cf-4a29028a98e7 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:22.112911 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76aa3874-2668-40b4-93cf-4a29028a98e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:22.113396 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc71b16-45f7-406e-b19f-946280d67b04 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:22.113638 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76aa3874-2668-40b4-93cf-4a29028a98e7 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:22.114854 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76aa3874-2668-40b4-93cf-4a29028a98e7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:22.116383 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76aa3874-2668-40b4-93cf-4a29028a98e7 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:22.118337 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 110/214] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:41:21 2022] GET /identity/v3/domains/default => generated 199 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:41:22.122540 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 105/215] 10.222.0.52 () {56 vars in 1101 bytes} [Sun Sep 25 00:41:21 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:22.123835 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76aa3874-2668-40b4-93cf-4a29028a98e7 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:22.133309 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc71b16-45f7-406e-b19f-946280d67b04 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:22.243233 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76aa3874-2668-40b4-93cf-4a29028a98e7 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:22.252144 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc71b16-45f7-406e-b19f-946280d67b04 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:22.280770 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6471a46-4f06-4419-8f8e-80e70bb775d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:22.281469 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6471a46-4f06-4419-8f8e-80e70bb775d2 None admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:22.281751 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6471a46-4f06-4419-8f8e-80e70bb775d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:22.282024 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6471a46-4f06-4419-8f8e-80e70bb775d2 None admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:22.282685 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6471a46-4f06-4419-8f8e-80e70bb775d2 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:22.284125 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6471a46-4f06-4419-8f8e-80e70bb775d2 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:22.285573 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:22.291205 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e676649-2c4a-494b-b419-2634e79288d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:22.291835 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e676649-2c4a-494b-b419-2634e79288d9 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:22.292111 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e676649-2c4a-494b-b419-2634e79288d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:22.292373 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e676649-2c4a-494b-b419-2634e79288d9 None admin] PATH_INFO: `/v3/users/8f4cf11f3bb6455fa917ce1410aa1fad` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:22.302295 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:22.308152 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 111/216] 10.222.0.52 () {60 vars in 1132 bytes} [Sun Sep 25 00:41:22 2022] POST /identity/v3/users => generated 297 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:41:22.308816 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e676649-2c4a-494b-b419-2634e79288d9 None admin] RBAC: Authorizing `identity:get_user(user_id=8f4cf11f3bb6455fa917ce1410aa1fad)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:22.309827 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04645460-29ef-4a84-94da-a973949b0627 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:22.310431 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04645460-29ef-4a84-94da-a973949b0627 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:22.311306 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04645460-29ef-4a84-94da-a973949b0627 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:22.312838 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 112/217] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:22 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:22.314222 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e676649-2c4a-494b-b419-2634e79288d9 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:22.316590 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 106/218] 10.222.0.52 () {56 vars in 1177 bytes} [Sun Sep 25 00:41:22 2022] GET /identity/v3/users/8f4cf11f3bb6455fa917ce1410aa1fad => generated 297 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:22.322806 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b5d2d6-d016-4a3e-88ca-45b37cb7d68c None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:22.323094 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b5d2d6-d016-4a3e-88ca-45b37cb7d68c None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:22.323372 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e676649-2c4a-494b-b419-2634e79288d9 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:22.323521 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b5d2d6-d016-4a3e-88ca-45b37cb7d68c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:22.344991 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-80b5d2d6-d016-4a3e-88ca-45b37cb7d68c None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:22.346421 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-80b5d2d6-d016-4a3e-88ca-45b37cb7d68c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:22.444234 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 113/219] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:41:22.474808 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e676649-2c4a-494b-b419-2634e79288d9 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:22.523582 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c4e6efe-cc03-4da7-add3-f1e6b2dd17cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:22.524126 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4e6efe-cc03-4da7-add3-f1e6b2dd17cd None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:22.524402 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4e6efe-cc03-4da7-add3-f1e6b2dd17cd None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:22.524669 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4e6efe-cc03-4da7-add3-f1e6b2dd17cd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:22.592116 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:22.603782 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c4e6efe-cc03-4da7-add3-f1e6b2dd17cd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:22.605573 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c4e6efe-cc03-4da7-add3-f1e6b2dd17cd None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:22.635199 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7677089-d2d7-4df2-9974-7ab609b3b9b9 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:22.635531 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7677089-d2d7-4df2-9974-7ab609b3b9b9 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:22.635737 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7677089-d2d7-4df2-9974-7ab609b3b9b9 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:22.637225 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 114/220] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:22.643025 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8089b4dd-721e-4ac4-ac8f-84fac0da9101 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:22.643307 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8089b4dd-721e-4ac4-ac8f-84fac0da9101 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:22.643582 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8089b4dd-721e-4ac4-ac8f-84fac0da9101 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:22.669443 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-8089b4dd-721e-4ac4-ac8f-84fac0da9101 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:22.670649 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-8089b4dd-721e-4ac4-ac8f-84fac0da9101 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:22.730328 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 107/221] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:41:22 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 408 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 00:41:22.738604 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c4e6efe-cc03-4da7-add3-f1e6b2dd17cd None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:22.769548 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 115/222] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 25 00:41:22.770419 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8089b4dd-721e-4ac4-ac8f-84fac0da9101 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:22.854637 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:22.879220 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c4e6efe-cc03-4da7-add3-f1e6b2dd17cd None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:22.889642 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:22.892711 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8089b4dd-721e-4ac4-ac8f-84fac0da9101 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:22.943819 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f823b1b-584d-43a1-bbcd-6474c5c2bc73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:22.944459 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f823b1b-584d-43a1-bbcd-6474c5c2bc73 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:22.944734 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f823b1b-584d-43a1-bbcd-6474c5c2bc73 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:22.945001 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f823b1b-584d-43a1-bbcd-6474c5c2bc73 None admin] PATH_INFO: `/v3/projects/86aea44da5fb4bdeb527c7ff257c5d43` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:22.948716 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f823b1b-584d-43a1-bbcd-6474c5c2bc73 None admin] RBAC: Authorizing `identity:get_project(project_id=86aea44da5fb4bdeb527c7ff257c5d43)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:22.950992 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f823b1b-584d-43a1-bbcd-6474c5c2bc73 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:22.952690 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 108/223] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:41:22 2022] GET /identity/v3/projects/86aea44da5fb4bdeb527c7ff257c5d43 => generated 302 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:41:22.957509 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3ebf106-72c4-4fb1-acb0-0cc769c06ab5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:22.958219 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ebf106-72c4-4fb1-acb0-0cc769c06ab5 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:22.958498 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ebf106-72c4-4fb1-acb0-0cc769c06ab5 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:22.958765 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ebf106-72c4-4fb1-acb0-0cc769c06ab5 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:22.959890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f823b1b-584d-43a1-bbcd-6474c5c2bc73 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:22.981708 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3ebf106-72c4-4fb1-acb0-0cc769c06ab5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:41:22.994952 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-a3ebf106-72c4-4fb1-acb0-0cc769c06ab5 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:41:22.994952 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:41:22.994952 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:41:22.994952 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:41:22.994952 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:41:22.994952 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:41:22.994952 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:41:22.994952 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:41:22.994952 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:41:22.994952 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:41:22.994952 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:41:22.994952 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:41:22.994952 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:41:22.994952 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:41:22.994952 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:41:22.994952 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:41:22.995897 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:41:22.995897 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:41:22.995897 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:41:22.995897 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:41:22.995897 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:41:22.995897 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:41:22.995897 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:41:22.995897 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:41:22.995897 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:41:22.995897 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:41:22.995897 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:41:22.995897 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:41:22.995897 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:41:22.995897 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:41:22.995897 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:41:22.995897 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:41:22.996696 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:41:22.996696 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:41:22.996696 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:41:22.996696 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:41:22.996696 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:41:22.996696 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:41:22.996696 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:41:22.996696 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:41:22.996696 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:41:22.996696 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:41:22.996696 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:41:22.996696 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:41:23.000132 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 116/224] 10.222.0.52 () {56 vars in 1096 bytes} [Sun Sep 25 00:41:22 2022] GET /identity/v3/roles/admin => generated 83 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:41:23.004560 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3ebf106-72c4-4fb1-acb0-0cc769c06ab5 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:23.132334 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f823b1b-584d-43a1-bbcd-6474c5c2bc73 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:23.176211 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1ca660-05b3-4505-9055-8239ff6f25cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:23.176830 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1ca660-05b3-4505-9055-8239ff6f25cc None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:23.177109 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1ca660-05b3-4505-9055-8239ff6f25cc None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:23.177430 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1ca660-05b3-4505-9055-8239ff6f25cc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:23.177845 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3ebf106-72c4-4fb1-acb0-0cc769c06ab5 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:23.178162 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d1ca660-05b3-4505-9055-8239ff6f25cc None admin] RBAC: Adding query filter params (role.id=40f54f5ac0994c51960ac413da8d2cc9, scope.project.id=86aea44da5fb4bdeb527c7ff257c5d43, user.id=8f4cf11f3bb6455fa917ce1410aa1fad) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:23.178471 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d1ca660-05b3-4505-9055-8239ff6f25cc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:23.181240 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d1ca660-05b3-4505-9055-8239ff6f25cc None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:23.188723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 109/225] 10.222.0.52 () {56 vars in 1374 bytes} [Sun Sep 25 00:41:22 2022] GET /identity/v3/role_assignments?user.id=8f4cf11f3bb6455fa917ce1410aa1fad&scope.project.id=86aea44da5fb4bdeb527c7ff257c5d43&role.id=40f54f5ac0994c51960ac413da8d2cc9 => generated 259 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:23.190159 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1ca660-05b3-4505-9055-8239ff6f25cc None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:23.199734 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:23.230868 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245010df-4718-4fdf-ad93-863566fb382c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:23.231501 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245010df-4718-4fdf-ad93-863566fb382c None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:23.231775 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245010df-4718-4fdf-ad93-863566fb382c None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:23.232042 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245010df-4718-4fdf-ad93-863566fb382c None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:23.232743 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-245010df-4718-4fdf-ad93-863566fb382c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:23.235464 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-245010df-4718-4fdf-ad93-863566fb382c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:23.236962 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-245010df-4718-4fdf-ad93-863566fb382c None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:23.240639 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:23.257091 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 117/226] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:41:23 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:41:23.264295 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245010df-4718-4fdf-ad93-863566fb382c None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:23.433798 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1ca660-05b3-4505-9055-8239ff6f25cc None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:23.458953 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245010df-4718-4fdf-ad93-863566fb382c None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:23.507672 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c2f3cfb-8617-4c76-ba58-ac3f8fa7111f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:23.508321 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2f3cfb-8617-4c76-ba58-ac3f8fa7111f None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:23.508599 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2f3cfb-8617-4c76-ba58-ac3f8fa7111f None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:23.508867 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2f3cfb-8617-4c76-ba58-ac3f8fa7111f None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:23.533275 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c2f3cfb-8617-4c76-ba58-ac3f8fa7111f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:41:23.535291 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f85e80d8-8af8-4d7a-8a60-1e06583f52d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:23.536156 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f85e80d8-8af8-4d7a-8a60-1e06583f52d8 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:23.536584 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f85e80d8-8af8-4d7a-8a60-1e06583f52d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:23.537012 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f85e80d8-8af8-4d7a-8a60-1e06583f52d8 None admin] PATH_INFO: `/v3/users/8ed437642b0742d4a84d4f8432538145` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:23.541123 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-0c2f3cfb-8617-4c76-ba58-ac3f8fa7111f None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 00:41:23.541123 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:41:23.541123 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:41:23.541123 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:41:23.541123 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:41:23.541123 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:41:23.541123 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:41:23.541123 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:41:23.541123 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:41:23.541123 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:41:23.541123 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:41:23.541123 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:41:23.541123 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:41:23.541123 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:41:23.541123 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:41:23.541123 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:41:23.542097 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:41:23.542097 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:41:23.542097 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:41:23.542097 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:41:23.542097 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:41:23.542097 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:41:23.542097 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:41:23.542097 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:41:23.542097 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:41:23.542097 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:41:23.542097 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:41:23.542097 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:41:23.542097 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:41:23.542097 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:41:23.542097 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:41:23.542097 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:41:23.542929 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:41:23.542929 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:41:23.542929 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:41:23.542929 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:41:23.542929 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:41:23.542929 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:41:23.542929 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:41:23.542929 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:41:23.542929 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:41:23.542929 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:41:23.542929 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 00:41:23.542929 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:41:23.542929 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 110/227] 10.222.0.52 () {56 vars in 1114 bytes} [Sun Sep 25 00:41:23 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 353 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:41:23.544567 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f85e80d8-8af8-4d7a-8a60-1e06583f52d8 None admin] RBAC: Authorizing `identity:get_user(user_id=8ed437642b0742d4a84d4f8432538145)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:23.549317 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f85e80d8-8af8-4d7a-8a60-1e06583f52d8 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:23.549317 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 118/228] 10.222.0.52 () {56 vars in 1177 bytes} [Sun Sep 25 00:41:23 2022] GET /identity/v3/users/8ed437642b0742d4a84d4f8432538145 => generated 283 bytes in 284 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:41:23.550562 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c2f3cfb-8617-4c76-ba58-ac3f8fa7111f None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:23.558405 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f85e80d8-8af8-4d7a-8a60-1e06583f52d8 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:23.665365 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:23.711743 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:23.735056 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c2f3cfb-8617-4c76-ba58-ac3f8fa7111f None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:23.796190 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f85e80d8-8af8-4d7a-8a60-1e06583f52d8 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:23.800669 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa0dfab-4e64-46e7-8cbc-9b3597c766df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:23.801305 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa0dfab-4e64-46e7-8cbc-9b3597c766df None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:23.801586 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa0dfab-4e64-46e7-8cbc-9b3597c766df None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:23.801853 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa0dfab-4e64-46e7-8cbc-9b3597c766df None admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:23.802526 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aa0dfab-4e64-46e7-8cbc-9b3597c766df None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:23.802827 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aa0dfab-4e64-46e7-8cbc-9b3597c766df None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:23.805367 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aa0dfab-4e64-46e7-8cbc-9b3597c766df None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:23.819681 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 111/229] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:41:23 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:23.826290 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa0dfab-4e64-46e7-8cbc-9b3597c766df None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:23.871596 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0568a38a-aff8-4486-8f6e-09b4bf96837e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:23.872305 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0568a38a-aff8-4486-8f6e-09b4bf96837e None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:23.872584 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0568a38a-aff8-4486-8f6e-09b4bf96837e None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:23.872852 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0568a38a-aff8-4486-8f6e-09b4bf96837e None admin] PATH_INFO: `/v3/projects/86aea44da5fb4bdeb527c7ff257c5d43` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:23.875182 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0568a38a-aff8-4486-8f6e-09b4bf96837e None admin] RBAC: Authorizing `identity:get_project(project_id=86aea44da5fb4bdeb527c7ff257c5d43)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:23.876644 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0568a38a-aff8-4486-8f6e-09b4bf96837e None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:23.878613 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 119/230] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:41:23 2022] GET /identity/v3/projects/86aea44da5fb4bdeb527c7ff257c5d43 => generated 302 bytes in 321 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:23.885109 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0568a38a-aff8-4486-8f6e-09b4bf96837e None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:24.024670 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0568a38a-aff8-4486-8f6e-09b4bf96837e None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:24.043707 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:24.068119 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3885beb-c776-48c6-9dc8-20e5f61095e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:24.068799 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3885beb-c776-48c6-9dc8-20e5f61095e1 None admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:24.069076 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3885beb-c776-48c6-9dc8-20e5f61095e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:24.069375 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3885beb-c776-48c6-9dc8-20e5f61095e1 None admin] PATH_INFO: `/v3/projects/86aea44da5fb4bdeb527c7ff257c5d43/users/8ed437642b0742d4a84d4f8432538145/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:24.071448 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3885beb-c776-48c6-9dc8-20e5f61095e1 None admin] RBAC: Authorizing `identity:create_grant(project_id=86aea44da5fb4bdeb527c7ff257c5d43, user_id=8ed437642b0742d4a84d4f8432538145, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:24.073067 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3885beb-c776-48c6-9dc8-20e5f61095e1 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:24.088884 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa0dfab-4e64-46e7-8cbc-9b3597c766df None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:24.091073 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 120/231] 10.222.0.52 () {58 vars in 1439 bytes} [Sun Sep 25 00:41:23 2022] PUT /identity/v3/projects/86aea44da5fb4bdeb527c7ff257c5d43/users/8ed437642b0742d4a84d4f8432538145/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:41:24.156034 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09fdb2ad-6a80-4bdf-b681-a41ce704778b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:24.156674 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09fdb2ad-6a80-4bdf-b681-a41ce704778b None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:24.156950 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09fdb2ad-6a80-4bdf-b681-a41ce704778b None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:24.157219 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09fdb2ad-6a80-4bdf-b681-a41ce704778b None admin] PATH_INFO: `/v3/users/8ed437642b0742d4a84d4f8432538145` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:24.159691 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09fdb2ad-6a80-4bdf-b681-a41ce704778b None admin] RBAC: Authorizing `identity:get_user(user_id=8ed437642b0742d4a84d4f8432538145)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:24.161214 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09fdb2ad-6a80-4bdf-b681-a41ce704778b None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:24.163166 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 112/232] 10.222.0.52 () {56 vars in 1177 bytes} [Sun Sep 25 00:41:23 2022] GET /identity/v3/users/8ed437642b0742d4a84d4f8432538145 => generated 283 bytes in 337 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:24.169276 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3885beb-c776-48c6-9dc8-20e5f61095e1 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:24.339380 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3885beb-c776-48c6-9dc8-20e5f61095e1 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:24.387320 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7022be88-07f5-4c97-98d9-57251d2a82c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:24.388007 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7022be88-07f5-4c97-98d9-57251d2a82c1 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:24.388278 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7022be88-07f5-4c97-98d9-57251d2a82c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:24.388543 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7022be88-07f5-4c97-98d9-57251d2a82c1 None admin] PATH_INFO: `/v3/projects/86aea44da5fb4bdeb527c7ff257c5d43` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:24.389776 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7022be88-07f5-4c97-98d9-57251d2a82c1 None admin] RBAC: Authorizing `identity:get_project(project_id=86aea44da5fb4bdeb527c7ff257c5d43)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:24.391226 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7022be88-07f5-4c97-98d9-57251d2a82c1 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:24.393326 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 121/233] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:41:24 2022] GET /identity/v3/projects/86aea44da5fb4bdeb527c7ff257c5d43 => generated 302 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:24.398635 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09fdb2ad-6a80-4bdf-b681-a41ce704778b None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:24.422807 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:24.547047 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09fdb2ad-6a80-4bdf-b681-a41ce704778b None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:24.583855 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c269be-509e-4205-bfa9-1b1fd0a89bdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:24.584515 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c269be-509e-4205-bfa9-1b1fd0a89bdd None admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:24.584788 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c269be-509e-4205-bfa9-1b1fd0a89bdd None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:24.585054 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c269be-509e-4205-bfa9-1b1fd0a89bdd None admin] PATH_INFO: `/v3/projects/86aea44da5fb4bdeb527c7ff257c5d43/users/8ed437642b0742d4a84d4f8432538145/roles/a917909d7aa84d00a841c77e7e7bdc8a` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:24.587423 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2c269be-509e-4205-bfa9-1b1fd0a89bdd None admin] RBAC: Authorizing `identity:create_grant(project_id=86aea44da5fb4bdeb527c7ff257c5d43, user_id=8ed437642b0742d4a84d4f8432538145, role_id=a917909d7aa84d00a841c77e7e7bdc8a)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:24.589629 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2c269be-509e-4205-bfa9-1b1fd0a89bdd None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:24.606813 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 113/234] 10.222.0.52 () {58 vars in 1439 bytes} [Sun Sep 25 00:41:24 2022] PUT /identity/v3/projects/86aea44da5fb4bdeb527c7ff257c5d43/users/8ed437642b0742d4a84d4f8432538145/roles/a917909d7aa84d00a841c77e7e7bdc8a => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:41:25.084391 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baca4506-9b3e-4926-8e6e-9f53d2e518fd None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:25.084913 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baca4506-9b3e-4926-8e6e-9f53d2e518fd None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:25.084913 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baca4506-9b3e-4926-8e6e-9f53d2e518fd None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:25.089309 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 122/235] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:25.095121 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75df6031-4c82-4e63-9bae-f7140847b171 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:25.095401 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75df6031-4c82-4e63-9bae-f7140847b171 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:25.095671 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75df6031-4c82-4e63-9bae-f7140847b171 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:25.120959 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-75df6031-4c82-4e63-9bae-f7140847b171 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:25.122266 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-75df6031-4c82-4e63-9bae-f7140847b171 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:25.222982 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 114/236] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:41:25.314204 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61fe9ee2-f097-4116-9d5a-69e05c7f4f63 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:25.314531 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61fe9ee2-f097-4116-9d5a-69e05c7f4f63 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:25.314734 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61fe9ee2-f097-4116-9d5a-69e05c7f4f63 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:25.316069 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 123/237] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:25.322378 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71c41c6-a425-4170-a940-8afe0cdfd54b None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:25.322658 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71c41c6-a425-4170-a940-8afe0cdfd54b None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:25.322927 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71c41c6-a425-4170-a940-8afe0cdfd54b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:25.331999 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:25.347115 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-e71c41c6-a425-4170-a940-8afe0cdfd54b None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:25.348230 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-e71c41c6-a425-4170-a940-8afe0cdfd54b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:25.448677 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 115/238] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:41:25.722175 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61fe9ee2-f097-4116-9d5a-69e05c7f4f63 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:25.781148 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:25.867792 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61fe9ee2-f097-4116-9d5a-69e05c7f4f63 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:25.923249 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bd7b412-8af6-4cb3-9885-c99a6516f807 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:25.924294 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd7b412-8af6-4cb3-9885-c99a6516f807 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:25.924824 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd7b412-8af6-4cb3-9885-c99a6516f807 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:25.925335 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd7b412-8af6-4cb3-9885-c99a6516f807 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:25.926764 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bd7b412-8af6-4cb3-9885-c99a6516f807 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:25.928439 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bd7b412-8af6-4cb3-9885-c99a6516f807 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:25.931767 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 124/239] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:41:25 2022] GET /identity/v3/domains/default => generated 199 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:25.938255 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e71c41c6-a425-4170-a940-8afe0cdfd54b None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:26.011867 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b9ddf3-d3df-41d4-a4d6-d28a22249f95 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:26.012487 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b9ddf3-d3df-41d4-a4d6-d28a22249f95 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:26.013204 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b9ddf3-d3df-41d4-a4d6-d28a22249f95 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:26.015224 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 125/240] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:26 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:26.024292 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c1ec37-623e-4d3d-8cd3-b7787de71e5c None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:26.024981 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c1ec37-623e-4d3d-8cd3-b7787de71e5c None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:26.025732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c1ec37-623e-4d3d-8cd3-b7787de71e5c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:26.054809 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-85c1ec37-623e-4d3d-8cd3-b7787de71e5c None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:26.056593 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-85c1ec37-623e-4d3d-8cd3-b7787de71e5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:26.139196 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e71c41c6-a425-4170-a940-8afe0cdfd54b None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:26.160180 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 126/241] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 25 00:41:26.161803 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c1ec37-623e-4d3d-8cd3-b7787de71e5c None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:26.176008 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:26.183131 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecbf2795-4294-4f13-8922-761fa635a2c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:26.184429 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecbf2795-4294-4f13-8922-761fa635a2c4 None admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:26.185034 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecbf2795-4294-4f13-8922-761fa635a2c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:26.185526 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecbf2795-4294-4f13-8922-761fa635a2c4 None admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:26.186368 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecbf2795-4294-4f13-8922-761fa635a2c4 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:26.188417 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecbf2795-4294-4f13-8922-761fa635a2c4 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:26.213878 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 116/242] 10.222.0.52 () {60 vars in 1132 bytes} [Sun Sep 25 00:41:25 2022] POST /identity/v3/users => generated 293 bytes in 276 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:41:26.264162 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:26.381358 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c1ec37-623e-4d3d-8cd3-b7787de71e5c None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:26.433886 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ca4da1-3aaf-44d0-b253-fd219190c98f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:26.434537 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ca4da1-3aaf-44d0-b253-fd219190c98f None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:26.434809 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ca4da1-3aaf-44d0-b253-fd219190c98f None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:26.435075 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ca4da1-3aaf-44d0-b253-fd219190c98f None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:26.459315 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ca4da1-3aaf-44d0-b253-fd219190c98f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:41:26.464853 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-33ca4da1-3aaf-44d0-b253-fd219190c98f None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 25 00:41:26.464853 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:41:26.464853 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:41:26.464853 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:41:26.464853 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:41:26.464853 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:41:26.464853 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:41:26.464853 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:41:26.464853 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:41:26.464853 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:41:26.464853 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:41:26.464853 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:41:26.464853 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:41:26.464853 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:41:26.464853 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:41:26.464853 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:41:26.465816 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:41:26.465816 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:41:26.465816 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:41:26.465816 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:41:26.465816 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:41:26.465816 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:41:26.465816 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:41:26.465816 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:41:26.465816 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:41:26.465816 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:41:26.465816 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:41:26.465816 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:41:26.465816 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:41:26.465816 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:41:26.465816 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:41:26.465816 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:41:26.467049 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:41:26.467049 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:41:26.467049 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:41:26.467049 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:41:26.467049 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:41:26.467049 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:41:26.467049 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:41:26.467049 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:41:26.467049 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:41:26.467049 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:41:26.467049 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 25 00:41:26.467049 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:41:26.467049 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 127/243] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:41:26 2022] GET /identity/v3/roles/reader => generated 84 bytes in 305 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:41:26.474599 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecbf2795-4294-4f13-8922-761fa635a2c4 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:26.599443 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecbf2795-4294-4f13-8922-761fa635a2c4 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:26.636204 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6a0895-50f2-4c8c-85ac-24934823d8eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:26.636813 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6a0895-50f2-4c8c-85ac-24934823d8eb None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:26.637086 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6a0895-50f2-4c8c-85ac-24934823d8eb None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:26.637384 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6a0895-50f2-4c8c-85ac-24934823d8eb None admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:26.638058 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6a0895-50f2-4c8c-85ac-24934823d8eb None admin] RBAC: Adding query filter params (name=reader) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:26.638360 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6a0895-50f2-4c8c-85ac-24934823d8eb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:26.639729 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6a0895-50f2-4c8c-85ac-24934823d8eb None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:26.648609 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 117/244] 10.222.0.52 () {56 vars in 1101 bytes} [Sun Sep 25 00:41:26 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:26.657490 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ca4da1-3aaf-44d0-b253-fd219190c98f None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:26.695350 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6a0895-50f2-4c8c-85ac-24934823d8eb None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:26.779293 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:26.809831 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ca4da1-3aaf-44d0-b253-fd219190c98f None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:26.812714 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:26.862214 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de715c9-aa4d-405f-aa81-47019f176819 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:26.862683 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6a0895-50f2-4c8c-85ac-24934823d8eb None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:26.863640 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de715c9-aa4d-405f-aa81-47019f176819 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:26.864188 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de715c9-aa4d-405f-aa81-47019f176819 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:26.864691 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de715c9-aa4d-405f-aa81-47019f176819 None admin] PATH_INFO: `/v3/users/e65955ff21b943b6bd4ba9a3195d316b` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:26.882939 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3de715c9-aa4d-405f-aa81-47019f176819 None admin] RBAC: Authorizing `identity:get_user(user_id=e65955ff21b943b6bd4ba9a3195d316b)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:26.886418 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3de715c9-aa4d-405f-aa81-47019f176819 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:26.889548 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 128/245] 10.222.0.52 () {56 vars in 1177 bytes} [Sun Sep 25 00:41:26 2022] GET /identity/v3/users/e65955ff21b943b6bd4ba9a3195d316b => generated 297 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:26.895860 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de715c9-aa4d-405f-aa81-47019f176819 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:26.908417 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6999ccb9-8584-4fd0-b9cc-779c4f054918 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:26.908957 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6999ccb9-8584-4fd0-b9cc-779c4f054918 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:26.909232 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6999ccb9-8584-4fd0-b9cc-779c4f054918 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:26.911715 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6999ccb9-8584-4fd0-b9cc-779c4f054918 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:26.932545 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6999ccb9-8584-4fd0-b9cc-779c4f054918 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:41:26.936143 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-6999ccb9-8584-4fd0-b9cc-779c4f054918 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 25 00:41:26.936143 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:41:26.936143 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:41:26.936143 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:41:26.936143 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:41:26.936143 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:41:26.936143 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:41:26.936143 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:41:26.936143 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:41:26.936143 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:41:26.936143 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:41:26.936143 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:41:26.936143 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:41:26.936143 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:41:26.936143 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:41:26.936143 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:41:26.937154 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:41:26.937154 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:41:26.937154 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:41:26.937154 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:41:26.937154 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:41:26.937154 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:41:26.937154 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:41:26.937154 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:41:26.937154 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:41:26.937154 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:41:26.937154 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:41:26.937154 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:41:26.937154 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:41:26.937154 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:41:26.937154 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:41:26.937154 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:41:26.938136 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:41:26.938136 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:41:26.938136 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:41:26.938136 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:41:26.938136 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:41:26.938136 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:41:26.938136 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:41:26.938136 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:41:26.938136 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:41:26.938136 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:41:26.938136 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 25 00:41:26.938136 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:41:26.939186 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 118/246] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:41:26 2022] GET /identity/v3/roles/member => generated 84 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:41:26.949690 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6999ccb9-8584-4fd0-b9cc-779c4f054918 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:27.049916 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de715c9-aa4d-405f-aa81-47019f176819 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:27.103287 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5769099a-fc73-4334-be6a-59ed1679c533 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:27.104256 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5769099a-fc73-4334-be6a-59ed1679c533 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:27.104618 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5769099a-fc73-4334-be6a-59ed1679c533 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:27.104892 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5769099a-fc73-4334-be6a-59ed1679c533 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:27.127491 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6999ccb9-8584-4fd0-b9cc-779c4f054918 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:27.140352 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:27.141001 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:27.171216 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8400a94e-b80f-4bbb-960f-a59f97524cd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:27.172168 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8400a94e-b80f-4bbb-960f-a59f97524cd4 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:27.172638 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8400a94e-b80f-4bbb-960f-a59f97524cd4 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:27.173098 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8400a94e-b80f-4bbb-960f-a59f97524cd4 None admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:27.174007 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8400a94e-b80f-4bbb-960f-a59f97524cd4 None admin] RBAC: Adding query filter params (name=member) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:27.174500 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8400a94e-b80f-4bbb-960f-a59f97524cd4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:27.176172 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8400a94e-b80f-4bbb-960f-a59f97524cd4 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:27.187279 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 119/247] 10.222.0.52 () {56 vars in 1101 bytes} [Sun Sep 25 00:41:26 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:27.188920 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf4b7df-5ca1-43b5-8b6c-3fd906835b29 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:27.189444 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf4b7df-5ca1-43b5-8b6c-3fd906835b29 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:27.189948 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf4b7df-5ca1-43b5-8b6c-3fd906835b29 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:27.191659 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 120/248] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:27.192104 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5769099a-fc73-4334-be6a-59ed1679c533 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:27.193699 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5769099a-fc73-4334-be6a-59ed1679c533 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:27.194631 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf4b7df-5ca1-43b5-8b6c-3fd906835b29 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:27.322086 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 129/249] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:41:26 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 427 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 00:41:27.324653 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df99e96f-d20c-4201-9e20-e412803cd521 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:27.325417 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df99e96f-d20c-4201-9e20-e412803cd521 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:27.325930 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df99e96f-d20c-4201-9e20-e412803cd521 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:27.328331 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf4b7df-5ca1-43b5-8b6c-3fd906835b29 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:27.361290 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-df99e96f-d20c-4201-9e20-e412803cd521 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:27.362531 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-df99e96f-d20c-4201-9e20-e412803cd521 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:27.402636 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c25b82b-3d71-4488-8343-1a803f4808e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:27.403200 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c25b82b-3d71-4488-8343-1a803f4808e6 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:27.403470 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c25b82b-3d71-4488-8343-1a803f4808e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:27.403730 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c25b82b-3d71-4488-8343-1a803f4808e6 None admin] PATH_INFO: `/v3/users/8f4cf11f3bb6455fa917ce1410aa1fad` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:27.404952 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c25b82b-3d71-4488-8343-1a803f4808e6 None admin] RBAC: Authorizing `identity:get_user(user_id=8f4cf11f3bb6455fa917ce1410aa1fad)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:27.406471 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c25b82b-3d71-4488-8343-1a803f4808e6 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:27.408290 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 121/250] 10.222.0.52 () {56 vars in 1177 bytes} [Sun Sep 25 00:41:27 2022] GET /identity/v3/users/8f4cf11f3bb6455fa917ce1410aa1fad => generated 297 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:41:27.409848 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c25b82b-3d71-4488-8343-1a803f4808e6 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:27.462145 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:27.483622 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 130/251] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:41:27.484026 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df99e96f-d20c-4201-9e20-e412803cd521 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:27.513592 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:27.652540 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c25b82b-3d71-4488-8343-1a803f4808e6 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:27.732734 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-958003e7-7542-4416-9b00-292134c64ed3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:27.733434 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-958003e7-7542-4416-9b00-292134c64ed3 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:27.733709 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-958003e7-7542-4416-9b00-292134c64ed3 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:27.733947 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-958003e7-7542-4416-9b00-292134c64ed3 None admin] PATH_INFO: `/v3/projects/86aea44da5fb4bdeb527c7ff257c5d43` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:27.735634 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-958003e7-7542-4416-9b00-292134c64ed3 None admin] RBAC: Authorizing `identity:get_project(project_id=86aea44da5fb4bdeb527c7ff257c5d43)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:27.737059 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-958003e7-7542-4416-9b00-292134c64ed3 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:27.740755 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 122/252] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:41:27 2022] GET /identity/v3/projects/86aea44da5fb4bdeb527c7ff257c5d43 => generated 302 bytes in 331 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:27.741931 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd84614-54b0-4881-ad71-73683de7194d None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:27.742206 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd84614-54b0-4881-ad71-73683de7194d None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:27.742474 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd84614-54b0-4881-ad71-73683de7194d None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:27.745017 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 123/253] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:27.747898 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd84614-54b0-4881-ad71-73683de7194d None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:27.752762 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df99e96f-d20c-4201-9e20-e412803cd521 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:27.817488 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d275711a-30a7-4125-9d2e-d3e536030ba4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:27.818122 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d275711a-30a7-4125-9d2e-d3e536030ba4 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:27.818391 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d275711a-30a7-4125-9d2e-d3e536030ba4 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:27.818652 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d275711a-30a7-4125-9d2e-d3e536030ba4 None admin] PATH_INFO: `/v3/projects/86aea44da5fb4bdeb527c7ff257c5d43` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:27.825730 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d275711a-30a7-4125-9d2e-d3e536030ba4 None admin] RBAC: Authorizing `identity:get_project(project_id=86aea44da5fb4bdeb527c7ff257c5d43)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:27.827212 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d275711a-30a7-4125-9d2e-d3e536030ba4 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:27.828961 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 131/254] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:41:27 2022] GET /identity/v3/projects/86aea44da5fb4bdeb527c7ff257c5d43 => generated 302 bytes in 345 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:27.830157 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0824ace4-f380-4097-91ef-14625a851f01 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:27.830431 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0824ace4-f380-4097-91ef-14625a851f01 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:27.830753 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0824ace4-f380-4097-91ef-14625a851f01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:27.859775 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-0824ace4-f380-4097-91ef-14625a851f01 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:27.861472 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-0824ace4-f380-4097-91ef-14625a851f01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:27.944228 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd84614-54b0-4881-ad71-73683de7194d None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:27.953662 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:27.961045 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:27.972978 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 132/255] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:41:27.973999 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0824ace4-f380-4097-91ef-14625a851f01 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:28.008293 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2a27b4f-ebf5-4488-b84c-6c49c3ef286c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:28.010193 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a27b4f-ebf5-4488-b84c-6c49c3ef286c None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:28.011856 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a27b4f-ebf5-4488-b84c-6c49c3ef286c None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:28.012282 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a27b4f-ebf5-4488-b84c-6c49c3ef286c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:28.013193 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2a27b4f-ebf5-4488-b84c-6c49c3ef286c None admin] RBAC: Adding query filter params (role.id=cc65bfe8ed2740b69427f705add475a5, scope.project.id=86aea44da5fb4bdeb527c7ff257c5d43, user.id=e65955ff21b943b6bd4ba9a3195d316b) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:28.014999 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2a27b4f-ebf5-4488-b84c-6c49c3ef286c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:28.016743 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2a27b4f-ebf5-4488-b84c-6c49c3ef286c None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:28.034726 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 124/256] 10.222.0.52 () {56 vars in 1374 bytes} [Sun Sep 25 00:41:27 2022] GET /identity/v3/role_assignments?user.id=e65955ff21b943b6bd4ba9a3195d316b&scope.project.id=86aea44da5fb4bdeb527c7ff257c5d43&role.id=cc65bfe8ed2740b69427f705add475a5 => generated 259 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:28.035874 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2a27b4f-ebf5-4488-b84c-6c49c3ef286c None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:28.157316 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0824ace4-f380-4097-91ef-14625a851f01 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:28.200157 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be88c046-a9de-4286-8d02-3d802fe35763 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:28.201976 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be88c046-a9de-4286-8d02-3d802fe35763 None admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:28.202394 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be88c046-a9de-4286-8d02-3d802fe35763 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:28.202799 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be88c046-a9de-4286-8d02-3d802fe35763 None admin] PATH_INFO: `/v3/projects/86aea44da5fb4bdeb527c7ff257c5d43/users/8f4cf11f3bb6455fa917ce1410aa1fad/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:28.205241 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be88c046-a9de-4286-8d02-3d802fe35763 None admin] RBAC: Authorizing `identity:create_grant(project_id=86aea44da5fb4bdeb527c7ff257c5d43, user_id=8f4cf11f3bb6455fa917ce1410aa1fad, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:28.206899 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be88c046-a9de-4286-8d02-3d802fe35763 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:28.234614 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2a27b4f-ebf5-4488-b84c-6c49c3ef286c None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:28.235280 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 133/257] 10.222.0.52 () {58 vars in 1439 bytes} [Sun Sep 25 00:41:27 2022] PUT /identity/v3/projects/86aea44da5fb4bdeb527c7ff257c5d43/users/8f4cf11f3bb6455fa917ce1410aa1fad/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 262 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:41:28.293702 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc4ba9a-e246-42b9-9eba-41d6c24132d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:28.294252 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc4ba9a-e246-42b9-9eba-41d6c24132d0 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:28.294536 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc4ba9a-e246-42b9-9eba-41d6c24132d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:28.294803 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc4ba9a-e246-42b9-9eba-41d6c24132d0 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:28.310502 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dc4ba9a-e246-42b9-9eba-41d6c24132d0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:41:28.316709 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-9dc4ba9a-e246-42b9-9eba-41d6c24132d0 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:41:28.316709 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:41:28.316709 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:41:28.316709 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:41:28.316709 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:41:28.316709 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:41:28.316709 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:41:28.316709 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:41:28.316709 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:41:28.316709 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:41:28.316709 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:41:28.316709 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:41:28.316709 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:41:28.316709 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:41:28.316709 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:41:28.316709 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:41:28.317671 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:41:28.317671 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:41:28.317671 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:41:28.317671 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:41:28.317671 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:41:28.317671 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:41:28.317671 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:41:28.317671 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:41:28.317671 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:41:28.317671 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:41:28.317671 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:41:28.317671 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:41:28.317671 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:41:28.317671 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:41:28.317671 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:41:28.317671 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:41:28.318490 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:41:28.318490 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:41:28.318490 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:41:28.318490 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:41:28.318490 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:41:28.318490 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:41:28.318490 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:41:28.318490 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:41:28.318490 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:41:28.318490 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:41:28.318490 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:41:28.318490 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:41:28.322137 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 125/258] 10.222.0.52 () {56 vars in 1096 bytes} [Sun Sep 25 00:41:28 2022] GET /identity/v3/roles/admin => generated 83 bytes in 286 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:41:28.324903 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be88c046-a9de-4286-8d02-3d802fe35763 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:28.410794 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:28.460938 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc4ba9a-e246-42b9-9eba-41d6c24132d0 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:28.505402 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be88c046-a9de-4286-8d02-3d802fe35763 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:28.515013 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:28.575515 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81fbf0b6-7530-4d32-9734-8f2b23c087e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:28.576070 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81fbf0b6-7530-4d32-9734-8f2b23c087e9 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:28.576342 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81fbf0b6-7530-4d32-9734-8f2b23c087e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:28.576627 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81fbf0b6-7530-4d32-9734-8f2b23c087e9 None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:28.577283 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81fbf0b6-7530-4d32-9734-8f2b23c087e9 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:28.577585 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81fbf0b6-7530-4d32-9734-8f2b23c087e9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:28.578964 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81fbf0b6-7530-4d32-9734-8f2b23c087e9 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:28.596800 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 134/259] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:41:28 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 272 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:28.603110 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc4ba9a-e246-42b9-9eba-41d6c24132d0 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:28.603364 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81fbf0b6-7530-4d32-9734-8f2b23c087e9 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:28.642563 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e815aa0-e271-4a9d-9f24-7b22252722fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:28.643552 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e815aa0-e271-4a9d-9f24-7b22252722fb None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:28.644020 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e815aa0-e271-4a9d-9f24-7b22252722fb None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:28.644526 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e815aa0-e271-4a9d-9f24-7b22252722fb None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:28.657715 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e815aa0-e271-4a9d-9f24-7b22252722fb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:41:28.663880 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-8e815aa0-e271-4a9d-9f24-7b22252722fb None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 00:41:28.663880 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:41:28.663880 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:41:28.663880 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:41:28.663880 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:41:28.663880 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:41:28.663880 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:41:28.663880 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:41:28.663880 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:41:28.663880 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:41:28.663880 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:41:28.663880 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:41:28.663880 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:41:28.663880 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:41:28.663880 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:41:28.663880 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:41:28.664853 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:41:28.664853 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:41:28.664853 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:41:28.664853 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:41:28.664853 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:41:28.664853 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:41:28.664853 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:41:28.664853 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:41:28.664853 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:41:28.664853 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:41:28.664853 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:41:28.664853 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:41:28.664853 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:41:28.664853 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:41:28.664853 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:41:28.664853 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:41:28.665858 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:41:28.665858 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:41:28.665858 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:41:28.665858 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:41:28.665858 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:41:28.665858 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:41:28.665858 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:41:28.665858 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:41:28.665858 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:41:28.665858 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:41:28.665858 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 00:41:28.665858 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:41:28.668701 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 126/260] 10.222.0.52 () {56 vars in 1114 bytes} [Sun Sep 25 00:41:28 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:41:28.678111 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e815aa0-e271-4a9d-9f24-7b22252722fb None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:28.745751 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81fbf0b6-7530-4d32-9734-8f2b23c087e9 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:28.758011 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:28.787112 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:28.791286 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77fa5e51-897f-4fc3-8111-c3c29b2231ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:28.792314 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77fa5e51-897f-4fc3-8111-c3c29b2231ce None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:28.792944 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77fa5e51-897f-4fc3-8111-c3c29b2231ce None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:28.793580 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77fa5e51-897f-4fc3-8111-c3c29b2231ce None admin] PATH_INFO: `/v3/users/8ed437642b0742d4a84d4f8432538145` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:28.795015 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77fa5e51-897f-4fc3-8111-c3c29b2231ce None admin] RBAC: Authorizing `identity:get_user(user_id=8ed437642b0742d4a84d4f8432538145)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:28.796739 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77fa5e51-897f-4fc3-8111-c3c29b2231ce None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:28.798657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e815aa0-e271-4a9d-9f24-7b22252722fb None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:28.799125 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 135/261] 10.222.0.52 () {56 vars in 1177 bytes} [Sun Sep 25 00:41:28 2022] GET /identity/v3/users/8ed437642b0742d4a84d4f8432538145 => generated 283 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:41:28.803990 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77fa5e51-897f-4fc3-8111-c3c29b2231ce None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:28.863950 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b45fec-c0a4-4b6d-b01f-e2870e5d8078 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:28.865105 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b45fec-c0a4-4b6d-b01f-e2870e5d8078 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:28.865608 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b45fec-c0a4-4b6d-b01f-e2870e5d8078 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:28.866084 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b45fec-c0a4-4b6d-b01f-e2870e5d8078 None admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:28.866976 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b45fec-c0a4-4b6d-b01f-e2870e5d8078 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:28.867511 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b45fec-c0a4-4b6d-b01f-e2870e5d8078 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:28.870561 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b45fec-c0a4-4b6d-b01f-e2870e5d8078 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:28.883243 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 127/262] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:41:28 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:41:28.889499 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b45fec-c0a4-4b6d-b01f-e2870e5d8078 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:28.966822 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77fa5e51-897f-4fc3-8111-c3c29b2231ce None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:29.041996 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-072b454c-995d-4ed4-a878-9307dcbdeb21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:29.043606 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072b454c-995d-4ed4-a878-9307dcbdeb21 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:29.043945 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072b454c-995d-4ed4-a878-9307dcbdeb21 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:29.044219 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072b454c-995d-4ed4-a878-9307dcbdeb21 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:29.105739 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b45fec-c0a4-4b6d-b01f-e2870e5d8078 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:29.158313 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:29.178731 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438bfe61-0176-4370-896c-bb09aa9446f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:29.179382 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438bfe61-0176-4370-896c-bb09aa9446f1 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:29.179655 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438bfe61-0176-4370-896c-bb09aa9446f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:29.179921 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438bfe61-0176-4370-896c-bb09aa9446f1 None admin] PATH_INFO: `/v3/users/8ed437642b0742d4a84d4f8432538145` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:29.181584 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438bfe61-0176-4370-896c-bb09aa9446f1 None admin] RBAC: Authorizing `identity:get_user(user_id=8ed437642b0742d4a84d4f8432538145)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:29.183055 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438bfe61-0176-4370-896c-bb09aa9446f1 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:29.184689 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 128/263] 10.222.0.52 () {56 vars in 1177 bytes} [Sun Sep 25 00:41:28 2022] GET /identity/v3/users/8ed437642b0742d4a84d4f8432538145 => generated 283 bytes in 296 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:41:29.191132 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438bfe61-0176-4370-896c-bb09aa9446f1 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:29.203037 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-072b454c-995d-4ed4-a878-9307dcbdeb21 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:29.204496 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-072b454c-995d-4ed4-a878-9307dcbdeb21 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:29.235194 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:29.328932 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 136/264] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:41:28 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 525 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 25 00:41:29.334142 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-072b454c-995d-4ed4-a878-9307dcbdeb21 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:29.445748 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:29.483284 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438bfe61-0176-4370-896c-bb09aa9446f1 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:29.485156 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-072b454c-995d-4ed4-a878-9307dcbdeb21 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:29.555595 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f971ae-efc2-4047-b738-40083de1110b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:29.556861 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f971ae-efc2-4047-b738-40083de1110b None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:29.557135 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f971ae-efc2-4047-b738-40083de1110b None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:29.557428 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f971ae-efc2-4047-b738-40083de1110b None admin] PATH_INFO: `/v3/projects/86aea44da5fb4bdeb527c7ff257c5d43` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:29.562351 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9f971ae-efc2-4047-b738-40083de1110b None admin] RBAC: Authorizing `identity:get_project(project_id=86aea44da5fb4bdeb527c7ff257c5d43)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:29.563820 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9f971ae-efc2-4047-b738-40083de1110b None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:29.565676 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 137/265] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:41:29 2022] GET /identity/v3/projects/86aea44da5fb4bdeb527c7ff257c5d43 => generated 302 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:41:29.572587 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f971ae-efc2-4047-b738-40083de1110b None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:29.584713 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0a9108b-565d-4b40-bd6b-c0cf260163c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:29.585385 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a9108b-565d-4b40-bd6b-c0cf260163c7 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:29.585664 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a9108b-565d-4b40-bd6b-c0cf260163c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:29.585932 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a9108b-565d-4b40-bd6b-c0cf260163c7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:29.704113 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0a9108b-565d-4b40-bd6b-c0cf260163c7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:29.705527 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0a9108b-565d-4b40-bd6b-c0cf260163c7 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:29.753633 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:29.767906 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f971ae-efc2-4047-b738-40083de1110b None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:29.819296 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490a26f0-bdaf-4796-b483-e42ad55f475b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:29.819914 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490a26f0-bdaf-4796-b483-e42ad55f475b None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:29.820186 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490a26f0-bdaf-4796-b483-e42ad55f475b None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:29.820446 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490a26f0-bdaf-4796-b483-e42ad55f475b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:29.821182 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-490a26f0-bdaf-4796-b483-e42ad55f475b None admin] RBAC: Adding query filter params (role.id=2c242abf6ed8410ab0a4a354ca1794fb, scope.project.id=86aea44da5fb4bdeb527c7ff257c5d43, user.id=8ed437642b0742d4a84d4f8432538145) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:29.821516 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-490a26f0-bdaf-4796-b483-e42ad55f475b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:29.824563 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-490a26f0-bdaf-4796-b483-e42ad55f475b None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:29.831854 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 138/266] 10.222.0.52 () {56 vars in 1374 bytes} [Sun Sep 25 00:41:29 2022] GET /identity/v3/role_assignments?user.id=8ed437642b0742d4a84d4f8432538145&scope.project.id=86aea44da5fb4bdeb527c7ff257c5d43&role.id=2c242abf6ed8410ab0a4a354ca1794fb => generated 608 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:41:29.833340 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-581c8339-ed9a-4841-a483-bd97f1f85bb9 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:29.833627 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-581c8339-ed9a-4841-a483-bd97f1f85bb9 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:29.834290 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-581c8339-ed9a-4841-a483-bd97f1f85bb9 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:29.835762 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 139/267] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:29.842626 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bfe86da-f081-49e2-a747-a1b9ee99d1a2 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:29.842955 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bfe86da-f081-49e2-a747-a1b9ee99d1a2 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:29.843162 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bfe86da-f081-49e2-a747-a1b9ee99d1a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:29.867401 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-7bfe86da-f081-49e2-a747-a1b9ee99d1a2 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:29.868820 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-7bfe86da-f081-49e2-a747-a1b9ee99d1a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:29.873099 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:29.940140 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 129/268] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:41:29 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 749 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 00:41:29.946029 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0a9108b-565d-4b40-bd6b-c0cf260163c7 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:30.015244 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 140/269] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:41:30.128537 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0a9108b-565d-4b40-bd6b-c0cf260163c7 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:30.164487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:30.178888 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb8164b-1567-4044-a0ad-db1f99d5714a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:30.179540 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb8164b-1567-4044-a0ad-db1f99d5714a None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:30.179816 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb8164b-1567-4044-a0ad-db1f99d5714a None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:30.180085 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb8164b-1567-4044-a0ad-db1f99d5714a None admin] PATH_INFO: `/v3/projects/86aea44da5fb4bdeb527c7ff257c5d43` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:30.186200 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb8164b-1567-4044-a0ad-db1f99d5714a None admin] RBAC: Authorizing `identity:get_project(project_id=86aea44da5fb4bdeb527c7ff257c5d43)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:30.187718 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb8164b-1567-4044-a0ad-db1f99d5714a None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:30.189532 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 130/270] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:41:29 2022] GET /identity/v3/projects/86aea44da5fb4bdeb527c7ff257c5d43 => generated 302 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:41:30.195718 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bfe86da-f081-49e2-a747-a1b9ee99d1a2 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:30.358256 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bfe86da-f081-49e2-a747-a1b9ee99d1a2 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:30.374451 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62caeda-cae1-4dde-a9aa-50729c345dbe None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:30.374777 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62caeda-cae1-4dde-a9aa-50729c345dbe None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:30.374995 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62caeda-cae1-4dde-a9aa-50729c345dbe None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:30.376601 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 131/271] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:30.384704 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4398bb53-c84b-4db9-b22f-91d44b768044 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:30.384704 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4398bb53-c84b-4db9-b22f-91d44b768044 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:30.384704 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4398bb53-c84b-4db9-b22f-91d44b768044 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:30.416422 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-4398bb53-c84b-4db9-b22f-91d44b768044 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:30.417704 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-4398bb53-c84b-4db9-b22f-91d44b768044 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:30.437192 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4d439a-9423-4477-a1d0-c10ea3241023 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:30.438209 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4d439a-9423-4477-a1d0-c10ea3241023 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:30.438685 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4d439a-9423-4477-a1d0-c10ea3241023 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:30.439260 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4d439a-9423-4477-a1d0-c10ea3241023 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:30.440304 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b4d439a-9423-4477-a1d0-c10ea3241023 None admin] RBAC: Adding query filter params (role.id=a917909d7aa84d00a841c77e7e7bdc8a, scope.project.id=86aea44da5fb4bdeb527c7ff257c5d43, user.id=8ed437642b0742d4a84d4f8432538145) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:30.440821 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b4d439a-9423-4477-a1d0-c10ea3241023 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:30.442483 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b4d439a-9423-4477-a1d0-c10ea3241023 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:30.454795 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 141/272] 10.222.0.52 () {56 vars in 1374 bytes} [Sun Sep 25 00:41:30 2022] GET /identity/v3/role_assignments?user.id=8ed437642b0742d4a84d4f8432538145&scope.project.id=86aea44da5fb4bdeb527c7ff257c5d43&role.id=a917909d7aa84d00a841c77e7e7bdc8a => generated 608 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:30.523650 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 132/273] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:41:30.526931 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4d439a-9423-4477-a1d0-c10ea3241023 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:30.534867 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:30.693082 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4d439a-9423-4477-a1d0-c10ea3241023 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:30.745999 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35668a22-f949-47e8-93cb-8a8811109ecc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:30.746646 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35668a22-f949-47e8-93cb-8a8811109ecc None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:30.746916 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35668a22-f949-47e8-93cb-8a8811109ecc None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:30.747180 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35668a22-f949-47e8-93cb-8a8811109ecc None admin] PATH_INFO: `/v3/domains/default` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:30.748360 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35668a22-f949-47e8-93cb-8a8811109ecc None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:30.749831 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35668a22-f949-47e8-93cb-8a8811109ecc None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:30.751759 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 142/274] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:41:30 2022] GET /identity/v3/domains/default => generated 199 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:30.756903 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4398bb53-c84b-4db9-b22f-91d44b768044 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:30.872212 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4398bb53-c84b-4db9-b22f-91d44b768044 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:30.894478 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:30.912368 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-181859c7-3ccb-472b-96b3-c880ea7e804c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:30.912979 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181859c7-3ccb-472b-96b3-c880ea7e804c None admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:30.913249 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181859c7-3ccb-472b-96b3-c880ea7e804c None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:30.913536 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181859c7-3ccb-472b-96b3-c880ea7e804c None admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:30.914188 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-181859c7-3ccb-472b-96b3-c880ea7e804c None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:30.915829 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-181859c7-3ccb-472b-96b3-c880ea7e804c None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:30.942295 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 133/275] 10.222.0.52 () {60 vars in 1132 bytes} [Sun Sep 25 00:41:30 2022] POST /identity/v3/users => generated 293 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:41:30.977950 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35668a22-f949-47e8-93cb-8a8811109ecc None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:31.067632 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:31.103034 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35668a22-f949-47e8-93cb-8a8811109ecc None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:31.156526 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2cecd0a-9b29-46de-9239-872157e863b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:31.156945 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2cecd0a-9b29-46de-9239-872157e863b1 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:31.157739 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2cecd0a-9b29-46de-9239-872157e863b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:31.157739 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2cecd0a-9b29-46de-9239-872157e863b1 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:31.174340 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2cecd0a-9b29-46de-9239-872157e863b1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:41:31.180443 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-a2cecd0a-9b29-46de-9239-872157e863b1 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 25 00:41:31.180443 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:41:31.180443 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:41:31.180443 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:41:31.180443 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:41:31.180443 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:41:31.180443 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:41:31.180443 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:41:31.180443 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:41:31.180443 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:41:31.180443 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:41:31.180443 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:41:31.180443 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:41:31.180443 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:41:31.180443 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:41:31.180443 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:41:31.181933 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:41:31.181933 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:41:31.181933 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:41:31.181933 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:41:31.181933 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:41:31.181933 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:41:31.181933 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:41:31.181933 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:41:31.181933 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:41:31.181933 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:41:31.181933 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:41:31.181933 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:41:31.181933 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:41:31.181933 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:41:31.181933 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:41:31.181933 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:41:31.182752 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:41:31.182752 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:41:31.182752 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:41:31.182752 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:41:31.182752 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:41:31.182752 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:41:31.182752 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:41:31.182752 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:41:31.182752 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:41:31.182752 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:41:31.182752 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 25 00:41:31.182752 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:41:31.185732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 143/276] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:41:30 2022] GET /identity/v3/roles/member => generated 84 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:41:31.192646 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-181859c7-3ccb-472b-96b3-c880ea7e804c None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:31.292816 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92087fb0-978a-40b1-a6b9-3b54645a8763 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:31.293111 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92087fb0-978a-40b1-a6b9-3b54645a8763 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:31.294230 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92087fb0-978a-40b1-a6b9-3b54645a8763 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:31.295789 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 144/277] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 00:41:31.301770 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ff56ba-bd74-45c2-aa7e-c1b88c6248ed None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:31.302011 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ff56ba-bd74-45c2-aa7e-c1b88c6248ed None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:31.302297 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ff56ba-bd74-45c2-aa7e-c1b88c6248ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:31.333988 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-c0ff56ba-bd74-45c2-aa7e-c1b88c6248ed None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:31.334913 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-c0ff56ba-bd74-45c2-aa7e-c1b88c6248ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:31.344207 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-181859c7-3ccb-472b-96b3-c880ea7e804c None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:31.390906 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab8fd72-2f06-4836-a7f2-8d5ceaca9958 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:31.391521 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab8fd72-2f06-4836-a7f2-8d5ceaca9958 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:31.391794 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab8fd72-2f06-4836-a7f2-8d5ceaca9958 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:31.392058 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab8fd72-2f06-4836-a7f2-8d5ceaca9958 None admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:31.392730 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ab8fd72-2f06-4836-a7f2-8d5ceaca9958 None admin] RBAC: Adding query filter params (name=member) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:31.393032 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ab8fd72-2f06-4836-a7f2-8d5ceaca9958 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:31.394611 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ab8fd72-2f06-4836-a7f2-8d5ceaca9958 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:31.404878 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 134/278] 10.222.0.52 () {56 vars in 1101 bytes} [Sun Sep 25 00:41:31 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:31.410553 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab8fd72-2f06-4836-a7f2-8d5ceaca9958 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:31.433394 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 145/279] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 25 00:41:31.435483 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875e2100-54f3-4984-b810-925c28664965 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:31.435976 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875e2100-54f3-4984-b810-925c28664965 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:31.436462 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875e2100-54f3-4984-b810-925c28664965 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:31.438300 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 146/280] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:31.445880 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ed0be1-d4fb-425e-b436-58b19a3bdbba None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:31.446209 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ed0be1-d4fb-425e-b436-58b19a3bdbba None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:31.447391 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ed0be1-d4fb-425e-b436-58b19a3bdbba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:31.458735 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:31.482690 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-65ed0be1-d4fb-425e-b436-58b19a3bdbba None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:31.487206 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-65ed0be1-d4fb-425e-b436-58b19a3bdbba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:31.555725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab8fd72-2f06-4836-a7f2-8d5ceaca9958 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:31.565981 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:31.579278 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 147/281] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:41:31.599294 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccfaa33c-7adf-4610-8720-9fcf63da111d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:31.600294 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccfaa33c-7adf-4610-8720-9fcf63da111d None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:31.600767 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccfaa33c-7adf-4610-8720-9fcf63da111d None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:31.601286 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccfaa33c-7adf-4610-8720-9fcf63da111d None admin] PATH_INFO: `/v3/users/83bef7d236294d53a1220c3822d6d64d` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:31.620990 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccfaa33c-7adf-4610-8720-9fcf63da111d None admin] RBAC: Authorizing `identity:get_user(user_id=83bef7d236294d53a1220c3822d6d64d)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:31.623049 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccfaa33c-7adf-4610-8720-9fcf63da111d None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:31.624968 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 135/282] 10.222.0.52 () {56 vars in 1177 bytes} [Sun Sep 25 00:41:31 2022] GET /identity/v3/users/83bef7d236294d53a1220c3822d6d64d => generated 293 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:41:31.630870 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ed0be1-d4fb-425e-b436-58b19a3bdbba None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:31.734571 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccfaa33c-7adf-4610-8720-9fcf63da111d None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:31.778097 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ed0be1-d4fb-425e-b436-58b19a3bdbba None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:31.847325 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0d1c6a-23aa-426e-ae92-2f7c770805f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:31.847853 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0d1c6a-23aa-426e-ae92-2f7c770805f3 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:31.848124 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0d1c6a-23aa-426e-ae92-2f7c770805f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:31.848384 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0d1c6a-23aa-426e-ae92-2f7c770805f3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:31.849181 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d0d1c6a-23aa-426e-ae92-2f7c770805f3 None admin] RBAC: Adding query filter params (role.id=40f54f5ac0994c51960ac413da8d2cc9, user.id=83bef7d236294d53a1220c3822d6d64d, scope.system=all) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:31.849510 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d0d1c6a-23aa-426e-ae92-2f7c770805f3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:31.850977 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d0d1c6a-23aa-426e-ae92-2f7c770805f3 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:31.864641 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 148/283] 10.222.0.52 () {56 vars in 1308 bytes} [Sun Sep 25 00:41:31 2022] GET /identity/v3/role_assignments?user.id=83bef7d236294d53a1220c3822d6d64d&scope.system=all&role.id=40f54f5ac0994c51960ac413da8d2cc9 => generated 226 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:31.905646 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:31.913649 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccfaa33c-7adf-4610-8720-9fcf63da111d None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:31.914295 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0d1c6a-23aa-426e-ae92-2f7c770805f3 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:31.984628 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a02a1f-76f8-44da-bd3b-634fbaa1431c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:31.985326 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a02a1f-76f8-44da-bd3b-634fbaa1431c None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:31.986437 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a02a1f-76f8-44da-bd3b-634fbaa1431c None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:31.986437 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a02a1f-76f8-44da-bd3b-634fbaa1431c None admin] PATH_INFO: `/v3/roles/member` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:31.998679 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:32.009872 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15a02a1f-76f8-44da-bd3b-634fbaa1431c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:41:32.017513 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-15a02a1f-76f8-44da-bd3b-634fbaa1431c None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 25 00:41:32.017513 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:41:32.017513 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:41:32.017513 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:41:32.017513 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:41:32.017513 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:41:32.017513 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:41:32.017513 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:41:32.017513 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:41:32.017513 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:41:32.017513 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:41:32.017513 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:41:32.017513 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:41:32.017513 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:41:32.017513 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:41:32.017513 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:41:32.018487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:41:32.018487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:41:32.018487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:41:32.018487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:41:32.018487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:41:32.018487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:41:32.018487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:41:32.018487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:41:32.018487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:41:32.018487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:41:32.018487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:41:32.018487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:41:32.018487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:41:32.018487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:41:32.018487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:41:32.018487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:41:32.019287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:41:32.019287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:41:32.019287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:41:32.019287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:41:32.019287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:41:32.019287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:41:32.019287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:41:32.019287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:41:32.019287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:41:32.019287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:41:32.019287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 25 00:41:32.019287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:41:32.019287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 136/284] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:41:31 2022] GET /identity/v3/roles/member => generated 84 bytes in 285 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:41:32.026205 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a02a1f-76f8-44da-bd3b-634fbaa1431c None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:32.099142 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0d1c6a-23aa-426e-ae92-2f7c770805f3 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:32.141416 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80921d31-3199-4806-b3bc-2d7478fadd64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:32.142048 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80921d31-3199-4806-b3bc-2d7478fadd64 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:32.142318 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80921d31-3199-4806-b3bc-2d7478fadd64 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:32.142582 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80921d31-3199-4806-b3bc-2d7478fadd64 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:32.154368 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80921d31-3199-4806-b3bc-2d7478fadd64 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:41:32.159334 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-80921d31-3199-4806-b3bc-2d7478fadd64 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 25 00:41:32.159334 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:41:32.159334 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:41:32.159334 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:41:32.159334 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:41:32.159334 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:41:32.159334 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:41:32.159334 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:41:32.159334 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:41:32.159334 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:41:32.159334 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:41:32.159334 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:41:32.159334 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:41:32.159334 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:41:32.159334 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:41:32.159334 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:41:32.161802 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:41:32.161802 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:41:32.161802 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:41:32.161802 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:41:32.161802 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:41:32.161802 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:41:32.161802 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:41:32.161802 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:41:32.161802 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:41:32.161802 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:41:32.161802 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:41:32.161802 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:41:32.161802 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:41:32.161802 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:41:32.161802 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:41:32.161802 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:41:32.163728 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:41:32.163728 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:41:32.163728 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:41:32.163728 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:41:32.163728 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:41:32.163728 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:41:32.163728 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:41:32.163728 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:41:32.163728 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:41:32.163728 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:41:32.163728 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 25 00:41:32.163728 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:41:32.163728 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 149/285] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:41:31 2022] GET /identity/v3/roles/reader => generated 84 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:41:32.173320 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80921d31-3199-4806-b3bc-2d7478fadd64 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:32.253311 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a02a1f-76f8-44da-bd3b-634fbaa1431c None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:32.280936 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80921d31-3199-4806-b3bc-2d7478fadd64 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:32.299134 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:32.320595 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07fd9c1c-d227-499b-94ac-57877934de01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:32.321210 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07fd9c1c-d227-499b-94ac-57877934de01 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:32.321684 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07fd9c1c-d227-499b-94ac-57877934de01 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:32.321952 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07fd9c1c-d227-499b-94ac-57877934de01 None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:32.322630 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07fd9c1c-d227-499b-94ac-57877934de01 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:32.322923 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07fd9c1c-d227-499b-94ac-57877934de01 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:32.324307 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07fd9c1c-d227-499b-94ac-57877934de01 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:32.328879 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3bda1ad-d788-4d13-95ad-eb1cfd28fdd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:32.329527 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bda1ad-d788-4d13-95ad-eb1cfd28fdd4 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:32.329805 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bda1ad-d788-4d13-95ad-eb1cfd28fdd4 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:32.330079 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bda1ad-d788-4d13-95ad-eb1cfd28fdd4 None admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:32.330760 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3bda1ad-d788-4d13-95ad-eb1cfd28fdd4 None admin] RBAC: Adding query filter params (name=member) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:32.331063 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3bda1ad-d788-4d13-95ad-eb1cfd28fdd4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:32.332471 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3bda1ad-d788-4d13-95ad-eb1cfd28fdd4 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:32.333885 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 150/286] 10.222.0.52 () {56 vars in 1101 bytes} [Sun Sep 25 00:41:32 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:41:32.340443 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07fd9c1c-d227-499b-94ac-57877934de01 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:32.352026 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 137/287] 10.222.0.52 () {56 vars in 1101 bytes} [Sun Sep 25 00:41:32 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 326 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:32.358334 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3bda1ad-d788-4d13-95ad-eb1cfd28fdd4 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:32.409636 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:32.473073 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07fd9c1c-d227-499b-94ac-57877934de01 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:32.512468 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84018126-1138-42d2-ba3c-e2d33a9e7c87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:32.513103 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84018126-1138-42d2-ba3c-e2d33a9e7c87 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:32.513507 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84018126-1138-42d2-ba3c-e2d33a9e7c87 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:32.513776 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84018126-1138-42d2-ba3c-e2d33a9e7c87 None admin] PATH_INFO: `/v3/users/e65955ff21b943b6bd4ba9a3195d316b` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:32.514989 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84018126-1138-42d2-ba3c-e2d33a9e7c87 None admin] RBAC: Authorizing `identity:get_user(user_id=e65955ff21b943b6bd4ba9a3195d316b)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:32.516477 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84018126-1138-42d2-ba3c-e2d33a9e7c87 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:32.518106 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 151/288] 10.222.0.52 () {56 vars in 1177 bytes} [Sun Sep 25 00:41:32 2022] GET /identity/v3/users/e65955ff21b943b6bd4ba9a3195d316b => generated 297 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:41:32.523993 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84018126-1138-42d2-ba3c-e2d33a9e7c87 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:32.578160 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:32.641668 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84018126-1138-42d2-ba3c-e2d33a9e7c87 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:32.645115 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3bda1ad-d788-4d13-95ad-eb1cfd28fdd4 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:32.686722 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c54b5b-3e84-4664-88cb-5996cab93145 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:32.687750 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c54b5b-3e84-4664-88cb-5996cab93145 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:32.688233 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c54b5b-3e84-4664-88cb-5996cab93145 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:32.688869 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c54b5b-3e84-4664-88cb-5996cab93145 None admin] PATH_INFO: `/v3/projects/86aea44da5fb4bdeb527c7ff257c5d43` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:32.692455 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3c54b5b-3e84-4664-88cb-5996cab93145 None admin] RBAC: Authorizing `identity:get_project(project_id=86aea44da5fb4bdeb527c7ff257c5d43)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:32.694531 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3c54b5b-3e84-4664-88cb-5996cab93145 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:32.703810 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 152/289] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:41:32 2022] GET /identity/v3/projects/86aea44da5fb4bdeb527c7ff257c5d43 => generated 302 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:32.710423 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c54b5b-3e84-4664-88cb-5996cab93145 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:32.727270 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52449aef-f830-48a3-b58c-003721832360 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:32.728824 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52449aef-f830-48a3-b58c-003721832360 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:32.729348 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52449aef-f830-48a3-b58c-003721832360 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:32.729800 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52449aef-f830-48a3-b58c-003721832360 None admin] PATH_INFO: `/v3/users/8f4cf11f3bb6455fa917ce1410aa1fad` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:32.736426 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52449aef-f830-48a3-b58c-003721832360 None admin] RBAC: Authorizing `identity:get_user(user_id=8f4cf11f3bb6455fa917ce1410aa1fad)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:32.738452 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52449aef-f830-48a3-b58c-003721832360 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:32.740305 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 138/290] 10.222.0.52 () {56 vars in 1177 bytes} [Sun Sep 25 00:41:32 2022] GET /identity/v3/users/8f4cf11f3bb6455fa917ce1410aa1fad => generated 297 bytes in 382 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:32.756027 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52449aef-f830-48a3-b58c-003721832360 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:32.883801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c54b5b-3e84-4664-88cb-5996cab93145 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:32.901407 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:32.917604 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:32.936642 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-817f8f27-c4de-4d8c-b40c-dcce60123744 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:32.937960 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817f8f27-c4de-4d8c-b40c-dcce60123744 None admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:32.938482 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817f8f27-c4de-4d8c-b40c-dcce60123744 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:32.938945 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817f8f27-c4de-4d8c-b40c-dcce60123744 None admin] PATH_INFO: `/v3/projects/86aea44da5fb4bdeb527c7ff257c5d43/users/e65955ff21b943b6bd4ba9a3195d316b/roles/cc65bfe8ed2740b69427f705add475a5` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:32.942517 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-817f8f27-c4de-4d8c-b40c-dcce60123744 None admin] RBAC: Authorizing `identity:create_grant(project_id=86aea44da5fb4bdeb527c7ff257c5d43, user_id=e65955ff21b943b6bd4ba9a3195d316b, role_id=cc65bfe8ed2740b69427f705add475a5)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:32.944274 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-817f8f27-c4de-4d8c-b40c-dcce60123744 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:32.949727 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52449aef-f830-48a3-b58c-003721832360 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:32.991317 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 153/291] 10.222.0.52 () {58 vars in 1439 bytes} [Sun Sep 25 00:41:32 2022] PUT /identity/v3/projects/86aea44da5fb4bdeb527c7ff257c5d43/users/e65955ff21b943b6bd4ba9a3195d316b/roles/cc65bfe8ed2740b69427f705add475a5 => generated 0 bytes in 277 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:41:33.033949 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf304ee7-dabc-4026-b7f0-efbd83e4f896 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:33.034962 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf304ee7-dabc-4026-b7f0-efbd83e4f896 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:33.035469 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf304ee7-dabc-4026-b7f0-efbd83e4f896 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:33.035942 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf304ee7-dabc-4026-b7f0-efbd83e4f896 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:33.138937 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf304ee7-dabc-4026-b7f0-efbd83e4f896 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:33.140856 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf304ee7-dabc-4026-b7f0-efbd83e4f896 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:33.262411 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:33.282852 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 139/292] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:41:32 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 527 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 25 00:41:33.288924 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-817f8f27-c4de-4d8c-b40c-dcce60123744 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:33.406812 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-817f8f27-c4de-4d8c-b40c-dcce60123744 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:33.445939 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7032e13-189c-4ebb-ba88-d87006bafafc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:33.447123 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7032e13-189c-4ebb-ba88-d87006bafafc None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:33.447400 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7032e13-189c-4ebb-ba88-d87006bafafc None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:33.447739 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7032e13-189c-4ebb-ba88-d87006bafafc None admin] PATH_INFO: `/v3/projects/86aea44da5fb4bdeb527c7ff257c5d43` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:33.448930 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7032e13-189c-4ebb-ba88-d87006bafafc None admin] RBAC: Authorizing `identity:get_project(project_id=86aea44da5fb4bdeb527c7ff257c5d43)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:33.450481 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7032e13-189c-4ebb-ba88-d87006bafafc None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:33.457155 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 154/293] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:41:33 2022] GET /identity/v3/projects/86aea44da5fb4bdeb527c7ff257c5d43 => generated 302 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:33.458653 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf304ee7-dabc-4026-b7f0-efbd83e4f896 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:33.586293 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf304ee7-dabc-4026-b7f0-efbd83e4f896 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:33.641617 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90520eba-f72d-4cbd-b3a2-e4574b19bf2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:33.642228 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90520eba-f72d-4cbd-b3a2-e4574b19bf2a None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:33.642504 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90520eba-f72d-4cbd-b3a2-e4574b19bf2a None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:33.642771 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90520eba-f72d-4cbd-b3a2-e4574b19bf2a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:33.643494 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90520eba-f72d-4cbd-b3a2-e4574b19bf2a None admin] RBAC: Adding query filter params (role.id=40f54f5ac0994c51960ac413da8d2cc9, scope.project.id=86aea44da5fb4bdeb527c7ff257c5d43, user.id=8f4cf11f3bb6455fa917ce1410aa1fad) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:33.643862 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90520eba-f72d-4cbd-b3a2-e4574b19bf2a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:33.645337 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90520eba-f72d-4cbd-b3a2-e4574b19bf2a None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:33.657436 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 140/294] 10.222.0.52 () {56 vars in 1374 bytes} [Sun Sep 25 00:41:33 2022] GET /identity/v3/role_assignments?user.id=8f4cf11f3bb6455fa917ce1410aa1fad&scope.project.id=86aea44da5fb4bdeb527c7ff257c5d43&role.id=40f54f5ac0994c51960ac413da8d2cc9 => generated 608 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:33.738342 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850471f3-bb44-46ef-ad0b-a065f28c2096 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:33.738859 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850471f3-bb44-46ef-ad0b-a065f28c2096 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:33.738932 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850471f3-bb44-46ef-ad0b-a065f28c2096 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:33.740566 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 155/295] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:33.746204 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fda69fa-37e6-44ba-afd4-e1898e3dff42 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:33.746489 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fda69fa-37e6-44ba-afd4-e1898e3dff42 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:33.746751 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fda69fa-37e6-44ba-afd4-e1898e3dff42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:33.775494 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-5fda69fa-37e6-44ba-afd4-e1898e3dff42 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:33.776704 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-5fda69fa-37e6-44ba-afd4-e1898e3dff42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:33.834772 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:33.859700 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 141/296] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:41:34.210249 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-850471f3-bb44-46ef-ad0b-a065f28c2096 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:34.263512 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:34.305412 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee06c41c-b6dd-475a-a530-61411127f7d4 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:34.306130 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee06c41c-b6dd-475a-a530-61411127f7d4 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:34.306287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee06c41c-b6dd-475a-a530-61411127f7d4 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:34.308361 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 142/297] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 00:41:34.315435 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a59286-bf35-4bfa-a99f-f8482d8e4131 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:34.316037 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a59286-bf35-4bfa-a99f-f8482d8e4131 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:34.316612 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a59286-bf35-4bfa-a99f-f8482d8e4131 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:34.337454 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-850471f3-bb44-46ef-ad0b-a065f28c2096 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:34.338469 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-19a59286-bf35-4bfa-a99f-f8482d8e4131 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:34.340023 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-19a59286-bf35-4bfa-a99f-f8482d8e4131 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:34.390433 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce32f23-fd31-44cb-b014-8778dcab2a61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:34.391076 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce32f23-fd31-44cb-b014-8778dcab2a61 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:34.391361 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce32f23-fd31-44cb-b014-8778dcab2a61 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:34.391633 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce32f23-fd31-44cb-b014-8778dcab2a61 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:34.412881 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce32f23-fd31-44cb-b014-8778dcab2a61 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:41:34.419994 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-fce32f23-fd31-44cb-b014-8778dcab2a61 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 25 00:41:34.419994 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:41:34.419994 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:41:34.419994 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:41:34.419994 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:41:34.419994 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:41:34.419994 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:41:34.419994 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:41:34.419994 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:41:34.419994 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:41:34.419994 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:41:34.419994 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:41:34.419994 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:41:34.419994 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:41:34.419994 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:41:34.419994 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:41:34.420937 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:41:34.420937 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:41:34.420937 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:41:34.420937 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:41:34.420937 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:41:34.420937 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:41:34.420937 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:41:34.420937 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:41:34.420937 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:41:34.420937 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:41:34.420937 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:41:34.420937 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:41:34.420937 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:41:34.420937 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:41:34.420937 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:41:34.420937 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:41:34.421787 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:41:34.421787 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:41:34.421787 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:41:34.421787 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:41:34.421787 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:41:34.421787 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:41:34.421787 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:41:34.421787 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:41:34.421787 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:41:34.421787 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:41:34.421787 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 25 00:41:34.421787 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:41:34.422397 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 156/298] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:41:34 2022] GET /identity/v3/roles/reader => generated 84 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:41:34.427431 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce32f23-fd31-44cb-b014-8778dcab2a61 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:34.432560 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 143/299] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:41:34.494848 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c12c1f6-cce3-401b-b108-f350aaa1471f None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:34.495156 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c12c1f6-cce3-401b-b108-f350aaa1471f None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:34.495392 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c12c1f6-cce3-401b-b108-f350aaa1471f None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:34.496772 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 144/300] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 00:41:34.504230 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d630bfc3-8dea-48fa-81e1-53ac006d8e9d None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:34.504644 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d630bfc3-8dea-48fa-81e1-53ac006d8e9d None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:34.505019 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d630bfc3-8dea-48fa-81e1-53ac006d8e9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:34.526351 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-d630bfc3-8dea-48fa-81e1-53ac006d8e9d None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:34.527715 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-d630bfc3-8dea-48fa-81e1-53ac006d8e9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:34.559000 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce32f23-fd31-44cb-b014-8778dcab2a61 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:34.560766 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:34.596412 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f57575-1c01-4cc6-a0d4-f6031d627170 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:34.597223 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f57575-1c01-4cc6-a0d4-f6031d627170 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:34.597627 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f57575-1c01-4cc6-a0d4-f6031d627170 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:34.597993 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f57575-1c01-4cc6-a0d4-f6031d627170 None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:34.598736 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24f57575-1c01-4cc6-a0d4-f6031d627170 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:34.599101 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24f57575-1c01-4cc6-a0d4-f6031d627170 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:34.600556 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24f57575-1c01-4cc6-a0d4-f6031d627170 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:34.609840 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 157/301] 10.222.0.52 () {56 vars in 1101 bytes} [Sun Sep 25 00:41:34 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:41:34.615766 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f57575-1c01-4cc6-a0d4-f6031d627170 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:34.616296 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 145/302] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 25 00:41:34.725536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f57575-1c01-4cc6-a0d4-f6031d627170 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:34.762749 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d630bfc3-8dea-48fa-81e1-53ac006d8e9d None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:34.773470 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e04b84f-3069-4673-8ca7-173d02212249 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:34.775274 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e04b84f-3069-4673-8ca7-173d02212249 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:34.775855 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e04b84f-3069-4673-8ca7-173d02212249 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:34.776380 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e04b84f-3069-4673-8ca7-173d02212249 None admin] PATH_INFO: `/v3/users/66ba69ef18a14430b6b22fe756146274` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:34.794953 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e04b84f-3069-4673-8ca7-173d02212249 None admin] RBAC: Authorizing `identity:get_user(user_id=66ba69ef18a14430b6b22fe756146274)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:34.796931 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e04b84f-3069-4673-8ca7-173d02212249 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:34.799409 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 158/303] 10.222.0.52 () {56 vars in 1177 bytes} [Sun Sep 25 00:41:34 2022] GET /identity/v3/users/66ba69ef18a14430b6b22fe756146274 => generated 293 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:34.805341 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e04b84f-3069-4673-8ca7-173d02212249 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:34.820319 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:34.834069 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:34.907908 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d630bfc3-8dea-48fa-81e1-53ac006d8e9d None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:34.928699 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e04b84f-3069-4673-8ca7-173d02212249 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:34.954297 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1df3bb66-358f-40f9-be7a-823f5d282820 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:34.955037 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df3bb66-358f-40f9-be7a-823f5d282820 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:34.955315 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df3bb66-358f-40f9-be7a-823f5d282820 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:34.955575 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df3bb66-358f-40f9-be7a-823f5d282820 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:34.956728 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1df3bb66-358f-40f9-be7a-823f5d282820 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:34.958263 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1df3bb66-358f-40f9-be7a-823f5d282820 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:34.960022 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 146/304] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:41:34 2022] GET /identity/v3/domains/default => generated 199 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:41:34.960597 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1df3bb66-358f-40f9-be7a-823f5d282820 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:34.981720 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79017c06-e16a-4d2a-8111-e0c1a06ee4a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:34.982327 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79017c06-e16a-4d2a-8111-e0c1a06ee4a5 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:34.982603 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79017c06-e16a-4d2a-8111-e0c1a06ee4a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:34.982868 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79017c06-e16a-4d2a-8111-e0c1a06ee4a5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:34.983586 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79017c06-e16a-4d2a-8111-e0c1a06ee4a5 None admin] RBAC: Adding query filter params (role.id=cc65bfe8ed2740b69427f705add475a5, user.id=66ba69ef18a14430b6b22fe756146274, scope.system=all) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:34.983887 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79017c06-e16a-4d2a-8111-e0c1a06ee4a5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:34.985347 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79017c06-e16a-4d2a-8111-e0c1a06ee4a5 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:34.991433 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 159/305] 10.222.0.52 () {56 vars in 1308 bytes} [Sun Sep 25 00:41:34 2022] GET /identity/v3/role_assignments?user.id=66ba69ef18a14430b6b22fe756146274&scope.system=all&role.id=cc65bfe8ed2740b69427f705add475a5 => generated 226 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:34.991867 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79017c06-e16a-4d2a-8111-e0c1a06ee4a5 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:35.117558 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1df3bb66-358f-40f9-be7a-823f5d282820 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:35.125769 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:35.137448 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:35.149846 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79017c06-e16a-4d2a-8111-e0c1a06ee4a5 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:35.174437 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56075ed6-7317-4a9b-b4ee-f13d7bfba5a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:35.175729 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56075ed6-7317-4a9b-b4ee-f13d7bfba5a8 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:35.176012 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56075ed6-7317-4a9b-b4ee-f13d7bfba5a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:35.176282 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56075ed6-7317-4a9b-b4ee-f13d7bfba5a8 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:35.191256 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56075ed6-7317-4a9b-b4ee-f13d7bfba5a8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:41:35.196349 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-56075ed6-7317-4a9b-b4ee-f13d7bfba5a8 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 25 00:41:35.196349 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:41:35.196349 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:41:35.196349 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:41:35.196349 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:41:35.196349 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:41:35.196349 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:41:35.196349 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:41:35.196349 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:41:35.196349 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:41:35.196349 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:41:35.196349 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:41:35.196349 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:41:35.196349 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:41:35.196349 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:41:35.196349 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:41:35.198287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:41:35.198287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:41:35.198287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:41:35.198287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:41:35.198287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:41:35.198287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:41:35.198287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:41:35.198287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:41:35.198287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:41:35.198287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:41:35.198287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:41:35.198287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:41:35.198287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:41:35.198287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:41:35.198287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:41:35.198287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:41:35.199084 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:41:35.199084 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:41:35.199084 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:41:35.199084 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:41:35.199084 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:41:35.199084 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:41:35.199084 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:41:35.199084 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:41:35.199084 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:41:35.199084 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:41:35.199084 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 25 00:41:35.199084 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:41:35.199691 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed4758ec-704c-454a-b162-63e520d6a510 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:35.199691 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed4758ec-704c-454a-b162-63e520d6a510 None admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:35.199839 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 147/306] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:41:34 2022] GET /identity/v3/roles/member => generated 84 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:41:35.199904 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed4758ec-704c-454a-b162-63e520d6a510 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:35.199904 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed4758ec-704c-454a-b162-63e520d6a510 None admin] PATH_INFO: `/v3/groups` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:35.199904 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed4758ec-704c-454a-b162-63e520d6a510 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:35.200991 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed4758ec-704c-454a-b162-63e520d6a510 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:35.209331 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56075ed6-7317-4a9b-b4ee-f13d7bfba5a8 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:35.219684 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 160/307] 10.222.0.52 () {60 vars in 1134 bytes} [Sun Sep 25 00:41:34 2022] POST /identity/v3/groups => generated 229 bytes in 228 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:41:35.344736 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56075ed6-7317-4a9b-b4ee-f13d7bfba5a8 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:35.386442 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dcd0456-be37-44bd-be61-e1e9f841fba4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:35.387090 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dcd0456-be37-44bd-be61-e1e9f841fba4 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:35.387373 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dcd0456-be37-44bd-be61-e1e9f841fba4 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:35.387646 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dcd0456-be37-44bd-be61-e1e9f841fba4 None admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:35.388369 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dcd0456-be37-44bd-be61-e1e9f841fba4 None admin] RBAC: Adding query filter params (name=member) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:35.388675 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dcd0456-be37-44bd-be61-e1e9f841fba4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:35.390164 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dcd0456-be37-44bd-be61-e1e9f841fba4 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:35.401293 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 148/308] 10.222.0.52 () {56 vars in 1101 bytes} [Sun Sep 25 00:41:35 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:35.407238 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed4758ec-704c-454a-b162-63e520d6a510 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:35.528642 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed4758ec-704c-454a-b162-63e520d6a510 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:35.566223 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41965d79-e0cc-45b8-b773-52cc27608fee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:35.566949 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41965d79-e0cc-45b8-b773-52cc27608fee None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:35.567234 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41965d79-e0cc-45b8-b773-52cc27608fee None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:35.567507 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41965d79-e0cc-45b8-b773-52cc27608fee None admin] PATH_INFO: `/v3/users/83bef7d236294d53a1220c3822d6d64d` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:35.568669 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41965d79-e0cc-45b8-b773-52cc27608fee None admin] RBAC: Authorizing `identity:get_user(user_id=83bef7d236294d53a1220c3822d6d64d)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:35.570142 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41965d79-e0cc-45b8-b773-52cc27608fee None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:35.572138 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 161/309] 10.222.0.52 () {56 vars in 1177 bytes} [Sun Sep 25 00:41:35 2022] GET /identity/v3/users/83bef7d236294d53a1220c3822d6d64d => generated 293 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:35.577875 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dcd0456-be37-44bd-be61-e1e9f841fba4 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:35.611205 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:35.703341 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dcd0456-be37-44bd-be61-e1e9f841fba4 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:35.746749 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-312ce0ad-d812-4a4a-8d24-649b23b07ab8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:35.747487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-312ce0ad-d812-4a4a-8d24-649b23b07ab8 None admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:35.747771 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-312ce0ad-d812-4a4a-8d24-649b23b07ab8 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:35.748046 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-312ce0ad-d812-4a4a-8d24-649b23b07ab8 None admin] PATH_INFO: `/v3/system/users/83bef7d236294d53a1220c3822d6d64d/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:35.749750 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-312ce0ad-d812-4a4a-8d24-649b23b07ab8 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=83bef7d236294d53a1220c3822d6d64d, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:35.751177 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-312ce0ad-d812-4a4a-8d24-649b23b07ab8 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:35.764339 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 149/310] 10.222.0.52 () {58 vars in 1334 bytes} [Sun Sep 25 00:41:35 2022] PUT /identity/v3/system/users/83bef7d236294d53a1220c3822d6d64d/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:41:35.851096 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8384f748-dae6-4b06-ae3a-10491a27a1bc None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:35.851612 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8384f748-dae6-4b06-ae3a-10491a27a1bc None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:35.851612 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8384f748-dae6-4b06-ae3a-10491a27a1bc None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:35.853769 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 162/311] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:35.859470 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9e5b0e-351f-4e85-8dc6-e8de916c36a7 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:35.859775 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9e5b0e-351f-4e85-8dc6-e8de916c36a7 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:35.860026 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9e5b0e-351f-4e85-8dc6-e8de916c36a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:35.880386 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-ad9e5b0e-351f-4e85-8dc6-e8de916c36a7 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:35.881649 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-ad9e5b0e-351f-4e85-8dc6-e8de916c36a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:35.969515 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 150/312] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:41:36.238433 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8384f748-dae6-4b06-ae3a-10491a27a1bc None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:36.257175 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:36.354982 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8384f748-dae6-4b06-ae3a-10491a27a1bc None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:36.390817 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b557d5-f69f-45d6-a3f7-eb3db7515b24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:36.391455 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b557d5-f69f-45d6-a3f7-eb3db7515b24 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:36.391732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b557d5-f69f-45d6-a3f7-eb3db7515b24 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:36.392002 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b557d5-f69f-45d6-a3f7-eb3db7515b24 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:36.403704 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19b557d5-f69f-45d6-a3f7-eb3db7515b24 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:41:36.409342 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-19b557d5-f69f-45d6-a3f7-eb3db7515b24 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 25 00:41:36.409342 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:41:36.409342 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:41:36.409342 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:41:36.409342 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:41:36.409342 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:41:36.409342 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:41:36.409342 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:41:36.409342 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:41:36.409342 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:41:36.409342 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:41:36.409342 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:41:36.409342 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:41:36.409342 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:41:36.409342 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:41:36.409342 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:41:36.410249 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:41:36.410249 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:41:36.410249 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:41:36.410249 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:41:36.410249 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:41:36.410249 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:41:36.410249 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:41:36.410249 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:41:36.410249 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:41:36.410249 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:41:36.410249 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:41:36.410249 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:41:36.410249 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:41:36.410249 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:41:36.410249 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:41:36.410249 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:41:36.411323 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:41:36.411323 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:41:36.411323 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:41:36.411323 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:41:36.411323 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:41:36.411323 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:41:36.411323 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:41:36.411323 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:41:36.411323 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:41:36.411323 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:41:36.411323 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 25 00:41:36.411323 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:41:36.411323 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 163/313] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:41:36 2022] GET /identity/v3/roles/reader => generated 84 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:41:36.416472 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9e5b0e-351f-4e85-8dc6-e8de916c36a7 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:36.444207 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:36.537940 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9e5b0e-351f-4e85-8dc6-e8de916c36a7 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:36.579556 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7089398-a179-4ed1-9f79-6a1a0401ada6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:36.580168 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7089398-a179-4ed1-9f79-6a1a0401ada6 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:36.580444 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7089398-a179-4ed1-9f79-6a1a0401ada6 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:36.580709 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7089398-a179-4ed1-9f79-6a1a0401ada6 None admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:36.581453 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7089398-a179-4ed1-9f79-6a1a0401ada6 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:36.581760 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7089398-a179-4ed1-9f79-6a1a0401ada6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:36.583176 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7089398-a179-4ed1-9f79-6a1a0401ada6 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:36.592694 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 151/314] 10.222.0.52 () {56 vars in 1101 bytes} [Sun Sep 25 00:41:36 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:41:36.598294 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b557d5-f69f-45d6-a3f7-eb3db7515b24 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:36.692080 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:36.720004 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b557d5-f69f-45d6-a3f7-eb3db7515b24 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:36.757153 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1176e233-8b67-4b55-a9ed-3a4a44048273 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:36.757824 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1176e233-8b67-4b55-a9ed-3a4a44048273 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:36.758101 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1176e233-8b67-4b55-a9ed-3a4a44048273 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:36.758370 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1176e233-8b67-4b55-a9ed-3a4a44048273 None admin] PATH_INFO: `/v3/users/e65955ff21b943b6bd4ba9a3195d316b` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:36.760217 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1176e233-8b67-4b55-a9ed-3a4a44048273 None admin] RBAC: Authorizing `identity:get_user(user_id=e65955ff21b943b6bd4ba9a3195d316b)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:36.761708 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1176e233-8b67-4b55-a9ed-3a4a44048273 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:36.763342 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 164/315] 10.222.0.52 () {56 vars in 1177 bytes} [Sun Sep 25 00:41:36 2022] GET /identity/v3/users/e65955ff21b943b6bd4ba9a3195d316b => generated 297 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:36.769508 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7089398-a179-4ed1-9f79-6a1a0401ada6 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:36.883692 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:36.905833 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7089398-a179-4ed1-9f79-6a1a0401ada6 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:36.942584 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8946fbc5-14e4-40b0-a81a-42c0f603f26d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:36.943201 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8946fbc5-14e4-40b0-a81a-42c0f603f26d None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:36.943478 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8946fbc5-14e4-40b0-a81a-42c0f603f26d None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:36.943750 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8946fbc5-14e4-40b0-a81a-42c0f603f26d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:37.021050 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8946fbc5-14e4-40b0-a81a-42c0f603f26d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:37.022561 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8946fbc5-14e4-40b0-a81a-42c0f603f26d None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:37.116540 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:37.135142 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 152/316] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:41:36 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 366 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 25 00:41:37.140601 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1176e233-8b67-4b55-a9ed-3a4a44048273 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:37.260217 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1176e233-8b67-4b55-a9ed-3a4a44048273 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:37.297324 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a81b7cfd-477d-4333-bfee-02676dbd762f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:37.298024 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a81b7cfd-477d-4333-bfee-02676dbd762f None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:37.298298 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a81b7cfd-477d-4333-bfee-02676dbd762f None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:37.298564 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a81b7cfd-477d-4333-bfee-02676dbd762f None admin] PATH_INFO: `/v3/projects/86aea44da5fb4bdeb527c7ff257c5d43` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:37.299801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a81b7cfd-477d-4333-bfee-02676dbd762f None admin] RBAC: Authorizing `identity:get_project(project_id=86aea44da5fb4bdeb527c7ff257c5d43)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:37.301384 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a81b7cfd-477d-4333-bfee-02676dbd762f None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:37.303502 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 165/317] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:41:37 2022] GET /identity/v3/projects/86aea44da5fb4bdeb527c7ff257c5d43 => generated 302 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:37.309729 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8946fbc5-14e4-40b0-a81a-42c0f603f26d None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:37.420284 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8946fbc5-14e4-40b0-a81a-42c0f603f26d None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:37.429706 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c3cf65-7ec1-4891-97b1-916e0cd61cff None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:37.430311 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c3cf65-7ec1-4891-97b1-916e0cd61cff None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:37.433215 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c3cf65-7ec1-4891-97b1-916e0cd61cff None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:37.435283 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 166/318] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:37 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 00:41:37.441097 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db62cadc-cecb-4e21-8f08-924e863e545e None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:37.441932 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db62cadc-cecb-4e21-8f08-924e863e545e None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:37.442503 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db62cadc-cecb-4e21-8f08-924e863e545e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:37.460475 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c79d16f-6448-4362-901e-2452dda3ac6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:37.461081 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c79d16f-6448-4362-901e-2452dda3ac6a None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:37.461385 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c79d16f-6448-4362-901e-2452dda3ac6a None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:37.461658 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c79d16f-6448-4362-901e-2452dda3ac6a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:37.462453 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c79d16f-6448-4362-901e-2452dda3ac6a None admin] RBAC: Adding query filter params (role.id=cc65bfe8ed2740b69427f705add475a5, scope.project.id=86aea44da5fb4bdeb527c7ff257c5d43, user.id=e65955ff21b943b6bd4ba9a3195d316b) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:37.462757 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c79d16f-6448-4362-901e-2452dda3ac6a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:37.464204 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c79d16f-6448-4362-901e-2452dda3ac6a None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:37.465826 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-db62cadc-cecb-4e21-8f08-924e863e545e None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:37.467471 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-db62cadc-cecb-4e21-8f08-924e863e545e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:37.470896 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:37.471153 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 153/319] 10.222.0.52 () {56 vars in 1374 bytes} [Sun Sep 25 00:41:37 2022] GET /identity/v3/role_assignments?user.id=e65955ff21b943b6bd4ba9a3195d316b&scope.project.id=86aea44da5fb4bdeb527c7ff257c5d43&role.id=cc65bfe8ed2740b69427f705add475a5 => generated 608 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:37.571446 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 167/320] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 25 00:41:37.645441 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e295e106-3f1b-4b13-8767-cc105e83936b None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:37.645790 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e295e106-3f1b-4b13-8767-cc105e83936b None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:37.645993 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e295e106-3f1b-4b13-8767-cc105e83936b None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:37.647632 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 154/321] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:37.655361 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a512088-8fbd-4255-b496-09893a3188f3 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:37.655653 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a512088-8fbd-4255-b496-09893a3188f3 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:37.655896 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a512088-8fbd-4255-b496-09893a3188f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:37.677498 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-0a512088-8fbd-4255-b496-09893a3188f3 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:37.678613 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-0a512088-8fbd-4255-b496-09893a3188f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:37.759071 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 168/322] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:41:37.861427 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e295e106-3f1b-4b13-8767-cc105e83936b None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:37.919612 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:37.984532 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e295e106-3f1b-4b13-8767-cc105e83936b None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:38.027139 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-775c8ecc-0f69-4510-a0a7-2b773b82519f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:38.027784 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775c8ecc-0f69-4510-a0a7-2b773b82519f None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:38.028057 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775c8ecc-0f69-4510-a0a7-2b773b82519f None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:38.028320 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775c8ecc-0f69-4510-a0a7-2b773b82519f None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:38.036545 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a512088-8fbd-4255-b496-09893a3188f3 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:38.045946 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-775c8ecc-0f69-4510-a0a7-2b773b82519f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:41:38.051107 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-775c8ecc-0f69-4510-a0a7-2b773b82519f None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 25 00:41:38.051107 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:41:38.051107 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:41:38.051107 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:41:38.051107 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:41:38.051107 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:41:38.051107 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:41:38.051107 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:41:38.051107 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:41:38.051107 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:41:38.051107 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:41:38.051107 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:41:38.051107 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:41:38.051107 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:41:38.051107 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:41:38.051107 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:41:38.052024 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:41:38.052024 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:41:38.052024 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:41:38.052024 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:41:38.052024 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:41:38.052024 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:41:38.052024 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:41:38.052024 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:41:38.052024 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:41:38.052024 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:41:38.052024 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:41:38.052024 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:41:38.052024 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:41:38.052024 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:41:38.052024 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:41:38.052024 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:41:38.052869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:41:38.052869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:41:38.052869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:41:38.052869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:41:38.052869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:41:38.052869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:41:38.052869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:41:38.052869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:41:38.052869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:41:38.052869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:41:38.052869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 25 00:41:38.052869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:41:38.052869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 155/323] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:41:37 2022] GET /identity/v3/roles/reader => generated 84 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:41:38.058406 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-775c8ecc-0f69-4510-a0a7-2b773b82519f None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:38.130589 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:38.162959 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a512088-8fbd-4255-b496-09893a3188f3 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:38.183711 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-775c8ecc-0f69-4510-a0a7-2b773b82519f None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:38.193885 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:38.198373 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8628beb8-e4b2-42b7-bb8d-308a07c21305 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:38.198991 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8628beb8-e4b2-42b7-bb8d-308a07c21305 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:38.199262 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8628beb8-e4b2-42b7-bb8d-308a07c21305 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:38.199581 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8628beb8-e4b2-42b7-bb8d-308a07c21305 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:38.200675 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8628beb8-e4b2-42b7-bb8d-308a07c21305 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:38.202131 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8628beb8-e4b2-42b7-bb8d-308a07c21305 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:38.203672 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 169/324] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:41:38 2022] GET /identity/v3/domains/default => generated 199 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:41:38.204761 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c057c14-2dfe-469d-8223-9a51f65d915b None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:38.205036 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c057c14-2dfe-469d-8223-9a51f65d915b None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:38.205370 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c057c14-2dfe-469d-8223-9a51f65d915b None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:38.209292 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 170/325] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:38 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:38.209292 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c057c14-2dfe-469d-8223-9a51f65d915b None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:38.223143 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba847f8-585a-46a5-b9d4-81ae47c954a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:38.223771 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba847f8-585a-46a5-b9d4-81ae47c954a6 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:38.224044 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba847f8-585a-46a5-b9d4-81ae47c954a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:38.224310 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba847f8-585a-46a5-b9d4-81ae47c954a6 None admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:38.224985 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba847f8-585a-46a5-b9d4-81ae47c954a6 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:38.225315 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba847f8-585a-46a5-b9d4-81ae47c954a6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:38.226719 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba847f8-585a-46a5-b9d4-81ae47c954a6 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:38.236862 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 156/326] 10.222.0.52 () {56 vars in 1101 bytes} [Sun Sep 25 00:41:38 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:41:38.238038 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8512c03a-c85d-4d47-a6f8-5d635f05254d None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:38.238579 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8512c03a-c85d-4d47-a6f8-5d635f05254d None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:38.238845 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8512c03a-c85d-4d47-a6f8-5d635f05254d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:38.257857 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-8512c03a-c85d-4d47-a6f8-5d635f05254d None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:38.259062 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-8512c03a-c85d-4d47-a6f8-5d635f05254d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:38.316573 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c057c14-2dfe-469d-8223-9a51f65d915b None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:38.328309 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 157/327] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:41:38.329147 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8512c03a-c85d-4d47-a6f8-5d635f05254d None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:38.354169 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6fbee0-b855-41c0-9c3d-58809e7f97f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:38.354778 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6fbee0-b855-41c0-9c3d-58809e7f97f9 None admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:38.355051 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6fbee0-b855-41c0-9c3d-58809e7f97f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:38.355314 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6fbee0-b855-41c0-9c3d-58809e7f97f9 None admin] PATH_INFO: `/v3/groups` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:38.355945 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d6fbee0-b855-41c0-9c3d-58809e7f97f9 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:38.357428 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d6fbee0-b855-41c0-9c3d-58809e7f97f9 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:38.369543 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:38.377115 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 171/328] 10.222.0.52 () {60 vars in 1134 bytes} [Sun Sep 25 00:41:38 2022] POST /identity/v3/groups => generated 226 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 00:41:38.460788 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8512c03a-c85d-4d47-a6f8-5d635f05254d None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:38.462703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:38.507702 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3326474f-b22e-4cdb-a9c4-152b5e5763a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:38.508277 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3326474f-b22e-4cdb-a9c4-152b5e5763a7 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:38.508547 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3326474f-b22e-4cdb-a9c4-152b5e5763a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:38.509457 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3326474f-b22e-4cdb-a9c4-152b5e5763a7 None admin] PATH_INFO: `/v3/users/66ba69ef18a14430b6b22fe756146274` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:38.511008 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3326474f-b22e-4cdb-a9c4-152b5e5763a7 None admin] RBAC: Authorizing `identity:get_user(user_id=66ba69ef18a14430b6b22fe756146274)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:38.512747 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3326474f-b22e-4cdb-a9c4-152b5e5763a7 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:38.519566 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 158/329] 10.222.0.52 () {56 vars in 1177 bytes} [Sun Sep 25 00:41:38 2022] GET /identity/v3/users/66ba69ef18a14430b6b22fe756146274 => generated 293 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:38.520861 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6fbee0-b855-41c0-9c3d-58809e7f97f9 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:38.608684 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3326474f-b22e-4cdb-a9c4-152b5e5763a7 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:38.648542 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6fbee0-b855-41c0-9c3d-58809e7f97f9 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:38.692113 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abbb12d7-74e7-4ff9-885d-af5eaf96d500 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:38.692807 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbb12d7-74e7-4ff9-885d-af5eaf96d500 None admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:38.693083 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbb12d7-74e7-4ff9-885d-af5eaf96d500 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:38.693379 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbb12d7-74e7-4ff9-885d-af5eaf96d500 None admin] PATH_INFO: `/v3/system/users/66ba69ef18a14430b6b22fe756146274/roles/cc65bfe8ed2740b69427f705add475a5` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:38.695133 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abbb12d7-74e7-4ff9-885d-af5eaf96d500 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=66ba69ef18a14430b6b22fe756146274, role_id=cc65bfe8ed2740b69427f705add475a5)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:38.698592 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abbb12d7-74e7-4ff9-885d-af5eaf96d500 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:38.708911 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 172/330] 10.222.0.52 () {58 vars in 1334 bytes} [Sun Sep 25 00:41:38 2022] PUT /identity/v3/system/users/66ba69ef18a14430b6b22fe756146274/roles/cc65bfe8ed2740b69427f705add475a5 => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:41:38.737452 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3326474f-b22e-4cdb-a9c4-152b5e5763a7 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:38.784590 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b53761a4-0000-4290-9f82-993e4c251954 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:38.785155 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53761a4-0000-4290-9f82-993e4c251954 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:38.785463 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53761a4-0000-4290-9f82-993e4c251954 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:38.785732 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53761a4-0000-4290-9f82-993e4c251954 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:38.802580 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b53761a4-0000-4290-9f82-993e4c251954 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:41:38.812982 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-b53761a4-0000-4290-9f82-993e4c251954 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 25 00:41:38.812982 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:41:38.812982 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:41:38.812982 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:41:38.812982 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:41:38.812982 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:41:38.812982 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:41:38.812982 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:41:38.812982 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:41:38.812982 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:41:38.812982 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:41:38.812982 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:41:38.812982 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:41:38.812982 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:41:38.812982 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:41:38.812982 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:41:38.814097 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:41:38.814097 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:41:38.814097 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:41:38.814097 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:41:38.814097 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:41:38.814097 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:41:38.814097 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:41:38.814097 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:41:38.814097 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:41:38.814097 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:41:38.814097 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:41:38.814097 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:41:38.814097 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:41:38.814097 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:41:38.814097 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:41:38.814097 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:41:38.814945 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:41:38.814945 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:41:38.814945 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:41:38.814945 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:41:38.814945 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:41:38.814945 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:41:38.814945 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:41:38.814945 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:41:38.814945 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:41:38.814945 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:41:38.814945 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 25 00:41:38.814945 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:41:38.816212 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 159/331] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:41:38 2022] GET /identity/v3/roles/member => generated 84 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:41:38.825500 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abbb12d7-74e7-4ff9-885d-af5eaf96d500 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:38.908092 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:38.967369 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abbb12d7-74e7-4ff9-885d-af5eaf96d500 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:39.008138 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d077c0-774d-4749-bffb-c62a351195f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:39.009588 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d077c0-774d-4749-bffb-c62a351195f2 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:39.009867 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d077c0-774d-4749-bffb-c62a351195f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:39.010129 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d077c0-774d-4749-bffb-c62a351195f2 None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:39.010854 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d077c0-774d-4749-bffb-c62a351195f2 None admin] RBAC: Adding query filter params (name=member) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:39.011157 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d077c0-774d-4749-bffb-c62a351195f2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:39.012912 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d077c0-774d-4749-bffb-c62a351195f2 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:39.022942 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 173/332] 10.222.0.52 () {56 vars in 1101 bytes} [Sun Sep 25 00:41:38 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:41:39.029899 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b53761a4-0000-4290-9f82-993e4c251954 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:39.070559 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:39.160542 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b53761a4-0000-4290-9f82-993e4c251954 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:39.203219 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c9d8779-4525-4234-b6cd-0e553170fa4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:39.203855 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9d8779-4525-4234-b6cd-0e553170fa4b None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:39.204189 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9d8779-4525-4234-b6cd-0e553170fa4b None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:39.204468 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9d8779-4525-4234-b6cd-0e553170fa4b None admin] PATH_INFO: `/v3/users/83bef7d236294d53a1220c3822d6d64d` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:39.205663 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c9d8779-4525-4234-b6cd-0e553170fa4b None admin] RBAC: Authorizing `identity:get_user(user_id=83bef7d236294d53a1220c3822d6d64d)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:39.207155 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c9d8779-4525-4234-b6cd-0e553170fa4b None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:39.209053 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 160/333] 10.222.0.52 () {56 vars in 1177 bytes} [Sun Sep 25 00:41:39 2022] GET /identity/v3/users/83bef7d236294d53a1220c3822d6d64d => generated 293 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:39.218406 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d077c0-774d-4749-bffb-c62a351195f2 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:39.353380 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d077c0-774d-4749-bffb-c62a351195f2 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:39.382520 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:39.398097 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c18a4ab1-255e-4a3d-9685-0b8bfaeddc69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:39.398734 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18a4ab1-255e-4a3d-9685-0b8bfaeddc69 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:39.399007 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18a4ab1-255e-4a3d-9685-0b8bfaeddc69 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:39.399274 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18a4ab1-255e-4a3d-9685-0b8bfaeddc69 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:39.400571 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c18a4ab1-255e-4a3d-9685-0b8bfaeddc69 None admin] RBAC: Adding query filter params (role.id=40f54f5ac0994c51960ac413da8d2cc9, user.id=83bef7d236294d53a1220c3822d6d64d, scope.system=all) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:39.400876 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c18a4ab1-255e-4a3d-9685-0b8bfaeddc69 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:39.402554 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c18a4ab1-255e-4a3d-9685-0b8bfaeddc69 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:39.416914 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 174/334] 10.222.0.52 () {56 vars in 1308 bytes} [Sun Sep 25 00:41:39 2022] GET /identity/v3/role_assignments?user.id=83bef7d236294d53a1220c3822d6d64d&scope.system=all&role.id=40f54f5ac0994c51960ac413da8d2cc9 => generated 510 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:41.089174 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9192ffd1-361a-4622-9ef2-eddad8f5a361 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:41.090246 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9192ffd1-361a-4622-9ef2-eddad8f5a361 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:41.090725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9192ffd1-361a-4622-9ef2-eddad8f5a361 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:41.092530 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 161/335] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:41 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:41.098501 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0652b304-8d03-4cfb-a470-f157a297e090 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:41.099379 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0652b304-8d03-4cfb-a470-f157a297e090 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:41.099843 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0652b304-8d03-4cfb-a470-f157a297e090 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:41.127994 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-0652b304-8d03-4cfb-a470-f157a297e090 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:41.129607 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-0652b304-8d03-4cfb-a470-f157a297e090 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:41.223847 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 175/336] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 25 00:41:41.253487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c0588d-bdcc-4858-99ff-d88791cd4ead None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:41.254088 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c0588d-bdcc-4858-99ff-d88791cd4ead None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:41.254605 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c0588d-bdcc-4858-99ff-d88791cd4ead None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:41.256137 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 162/337] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:41.262375 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6268c093-916d-44a8-917e-ec57bcb44004 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:41.262985 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6268c093-916d-44a8-917e-ec57bcb44004 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:41.263517 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6268c093-916d-44a8-917e-ec57bcb44004 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:41.290518 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-6268c093-916d-44a8-917e-ec57bcb44004 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:41.292348 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-6268c093-916d-44a8-917e-ec57bcb44004 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:41.384393 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 176/338] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:41:41.663806 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b20815-e429-4a3b-b0ab-18c37fd4f342 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:41.664401 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b20815-e429-4a3b-b0ab-18c37fd4f342 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:41.664929 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b20815-e429-4a3b-b0ab-18c37fd4f342 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:41.666786 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 163/339] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:41.673517 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0abe3ed-72c4-471b-8705-6d9bc6621e2e None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:41.674081 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0abe3ed-72c4-471b-8705-6d9bc6621e2e None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:41.674548 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0abe3ed-72c4-471b-8705-6d9bc6621e2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:41.693816 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-e0abe3ed-72c4-471b-8705-6d9bc6621e2e None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:41.695341 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-e0abe3ed-72c4-471b-8705-6d9bc6621e2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:41.735726 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:41.775931 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 177/340] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 25 00:41:41.866999 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62b20815-e429-4a3b-b0ab-18c37fd4f342 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:41.873447 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0abe3ed-72c4-471b-8705-6d9bc6621e2e None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:42.042996 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:42.049958 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62b20815-e429-4a3b-b0ab-18c37fd4f342 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:42.066081 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0abe3ed-72c4-471b-8705-6d9bc6621e2e None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:42.118483 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-495f7789-e3ab-4941-beed-e35f1b9419ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:42.119226 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495f7789-e3ab-4941-beed-e35f1b9419ae None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:42.119511 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495f7789-e3ab-4941-beed-e35f1b9419ae None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:42.119779 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495f7789-e3ab-4941-beed-e35f1b9419ae None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:42.121979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c77d9070-cdce-4cd2-b06d-2aef971e1e79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:42.122621 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77d9070-cdce-4cd2-b06d-2aef971e1e79 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:42.122890 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77d9070-cdce-4cd2-b06d-2aef971e1e79 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:42.123152 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77d9070-cdce-4cd2-b06d-2aef971e1e79 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:42.139234 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c77d9070-cdce-4cd2-b06d-2aef971e1e79 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:41:42.140254 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-495f7789-e3ab-4941-beed-e35f1b9419ae None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:41:42.144673 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-c77d9070-cdce-4cd2-b06d-2aef971e1e79 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 25 00:41:42.144673 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:41:42.144673 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:41:42.144673 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:41:42.144673 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:41:42.144673 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:41:42.144673 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:41:42.144673 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:41:42.144673 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:41:42.144673 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:41:42.144673 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:41:42.144673 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:41:42.144673 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:41:42.144673 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:41:42.144673 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:41:42.144673 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:41:42.150176 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:41:42.150176 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:41:42.150176 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:41:42.150176 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:41:42.150176 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:41:42.150176 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:41:42.150176 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:41:42.150176 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:41:42.150176 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:41:42.150176 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:41:42.150176 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:41:42.150176 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:41:42.150176 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:41:42.150176 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:41:42.150176 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:41:42.150176 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:41:42.151418 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:41:42.151418 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:41:42.151418 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:41:42.151418 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:41:42.151418 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:41:42.151418 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:41:42.151418 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:41:42.151418 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:41:42.151418 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:41:42.151418 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:41:42.151418 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 25 00:41:42.151418 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:41:42.154157 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 178/341] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:41:41 2022] GET /identity/v3/roles/member => generated 84 bytes in 281 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:41:42.154785 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-495f7789-e3ab-4941-beed-e35f1b9419ae None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 25 00:41:42.154785 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:41:42.154785 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:41:42.154785 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:41:42.154785 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:41:42.154785 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:41:42.154785 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:41:42.154785 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:41:42.154785 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:41:42.154785 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:41:42.154785 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:41:42.154785 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:41:42.154785 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:41:42.154785 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:41:42.154785 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:41:42.154785 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:41:42.155640 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:41:42.155640 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:41:42.155640 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:41:42.155640 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:41:42.155640 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:41:42.155640 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:41:42.155640 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:41:42.155640 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:41:42.155640 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:41:42.155640 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:41:42.155640 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:41:42.155640 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:41:42.155640 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:41:42.155640 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:41:42.155640 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:41:42.155640 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:41:42.156422 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:41:42.156422 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:41:42.156422 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:41:42.156422 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:41:42.156422 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:41:42.156422 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:41:42.156422 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:41:42.156422 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:41:42.156422 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:41:42.156422 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:41:42.156422 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 25 00:41:42.156422 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:41:42.156422 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 164/342] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:41:41 2022] GET /identity/v3/roles/reader => generated 84 bytes in 290 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:41:42.161700 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c77d9070-cdce-4cd2-b06d-2aef971e1e79 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:42.167341 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-495f7789-e3ab-4941-beed-e35f1b9419ae None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:42.206254 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:42.325721 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c77d9070-cdce-4cd2-b06d-2aef971e1e79 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:42.396948 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83cab2be-2ce6-40b4-8ddf-0f142fc16d48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:42.397993 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83cab2be-2ce6-40b4-8ddf-0f142fc16d48 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:42.398420 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83cab2be-2ce6-40b4-8ddf-0f142fc16d48 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:42.398838 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83cab2be-2ce6-40b4-8ddf-0f142fc16d48 None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:42.400490 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83cab2be-2ce6-40b4-8ddf-0f142fc16d48 None admin] RBAC: Adding query filter params (name=member) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:42.401033 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83cab2be-2ce6-40b4-8ddf-0f142fc16d48 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:42.403438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83cab2be-2ce6-40b4-8ddf-0f142fc16d48 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:42.435055 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 179/343] 10.222.0.52 () {56 vars in 1101 bytes} [Sun Sep 25 00:41:42 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:42.438034 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-495f7789-e3ab-4941-beed-e35f1b9419ae None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:42.440976 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83cab2be-2ce6-40b4-8ddf-0f142fc16d48 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:42.505136 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7741d94e-092c-442d-9b77-99441ea162e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:42.505773 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7741d94e-092c-442d-9b77-99441ea162e7 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:42.506049 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7741d94e-092c-442d-9b77-99441ea162e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:42.506313 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7741d94e-092c-442d-9b77-99441ea162e7 None admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:42.506983 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7741d94e-092c-442d-9b77-99441ea162e7 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:42.507281 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7741d94e-092c-442d-9b77-99441ea162e7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:42.509191 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7741d94e-092c-442d-9b77-99441ea162e7 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:42.518789 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 165/344] 10.222.0.52 () {56 vars in 1101 bytes} [Sun Sep 25 00:41:42 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 352 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:41:42.520152 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7965e969-8697-453c-889f-3f814bf51701 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:42.520434 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7965e969-8697-453c-889f-3f814bf51701 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:42.520706 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7965e969-8697-453c-889f-3f814bf51701 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:42.526685 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 166/345] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:42 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:42.527318 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7965e969-8697-453c-889f-3f814bf51701 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:42.558920 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:42.607359 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83cab2be-2ce6-40b4-8ddf-0f142fc16d48 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:42.621681 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:42.652321 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea15d69-ecfc-494e-8e68-3622695ce4f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:42.652963 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea15d69-ecfc-494e-8e68-3622695ce4f0 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:42.653240 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea15d69-ecfc-494e-8e68-3622695ce4f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:42.653541 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea15d69-ecfc-494e-8e68-3622695ce4f0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:42.664776 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7965e969-8697-453c-889f-3f814bf51701 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:42.710786 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b847b91-f727-4722-b44d-71e607b66df0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:42.711437 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b847b91-f727-4722-b44d-71e607b66df0 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:42.711743 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b847b91-f727-4722-b44d-71e607b66df0 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:42.712230 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b847b91-f727-4722-b44d-71e607b66df0 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:42.723777 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b847b91-f727-4722-b44d-71e607b66df0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:41:42.731349 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-3b847b91-f727-4722-b44d-71e607b66df0 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 00:41:42.731349 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:41:42.731349 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:41:42.731349 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:41:42.731349 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:41:42.731349 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:41:42.731349 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:41:42.731349 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:41:42.731349 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:41:42.731349 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:41:42.731349 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:41:42.731349 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:41:42.731349 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:41:42.731349 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:41:42.731349 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:41:42.731349 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:41:42.732650 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:41:42.732650 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:41:42.732650 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:41:42.732650 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:41:42.732650 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:41:42.732650 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:41:42.732650 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:41:42.732650 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:41:42.732650 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:41:42.732650 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:41:42.732650 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:41:42.732650 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:41:42.732650 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:41:42.732650 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:41:42.732650 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:41:42.732650 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:41:42.733781 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:41:42.733781 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:41:42.733781 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:41:42.733781 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:41:42.733781 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:41:42.733781 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:41:42.733781 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:41:42.733781 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:41:42.733781 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:41:42.733781 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:41:42.733781 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 00:41:42.733781 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:41:42.736137 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 167/346] 10.222.0.52 () {56 vars in 1114 bytes} [Sun Sep 25 00:41:42 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:41:42.737160 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b847b91-f727-4722-b44d-71e607b66df0 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:42.744793 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ea15d69-ecfc-494e-8e68-3622695ce4f0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:42.746391 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ea15d69-ecfc-494e-8e68-3622695ce4f0 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:42.832983 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:42.855450 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b847b91-f727-4722-b44d-71e607b66df0 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:42.858994 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:42.863563 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 180/347] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:41:42 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 423 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 25 00:41:42.870843 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adee03d3-926f-4c84-9837-d4c5e2529093 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:42.870843 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adee03d3-926f-4c84-9837-d4c5e2529093 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:42.870843 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adee03d3-926f-4c84-9837-d4c5e2529093 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:42.890224 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-adee03d3-926f-4c84-9837-d4c5e2529093 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:42.891725 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-adee03d3-926f-4c84-9837-d4c5e2529093 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:42.901745 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4116853f-8ee4-4ab0-85c8-3e025db1ded0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:42.902682 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4116853f-8ee4-4ab0-85c8-3e025db1ded0 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:42.903154 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4116853f-8ee4-4ab0-85c8-3e025db1ded0 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:42.903432 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4116853f-8ee4-4ab0-85c8-3e025db1ded0 None admin] PATH_INFO: `/v3/users/66ba69ef18a14430b6b22fe756146274` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:42.904738 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4116853f-8ee4-4ab0-85c8-3e025db1ded0 None admin] RBAC: Authorizing `identity:get_user(user_id=66ba69ef18a14430b6b22fe756146274)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:42.906429 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4116853f-8ee4-4ab0-85c8-3e025db1ded0 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:42.908348 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 168/348] 10.222.0.52 () {56 vars in 1177 bytes} [Sun Sep 25 00:41:42 2022] GET /identity/v3/users/66ba69ef18a14430b6b22fe756146274 => generated 293 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:42.908821 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4116853f-8ee4-4ab0-85c8-3e025db1ded0 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:42.978317 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 181/349] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:41:42.979654 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adee03d3-926f-4c84-9837-d4c5e2529093 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:43.049572 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4116853f-8ee4-4ab0-85c8-3e025db1ded0 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:43.095820 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adee03d3-926f-4c84-9837-d4c5e2529093 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:43.126715 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:43.127317 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ee8072-1e38-4713-baba-450ada01b394 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:43.127916 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ee8072-1e38-4713-baba-450ada01b394 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:43.129184 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ee8072-1e38-4713-baba-450ada01b394 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:43.129478 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ee8072-1e38-4713-baba-450ada01b394 None admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:43.130293 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8ee8072-1e38-4713-baba-450ada01b394 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:43.130596 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8ee8072-1e38-4713-baba-450ada01b394 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:43.132631 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8ee8072-1e38-4713-baba-450ada01b394 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:43.145482 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 169/350] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:41:42 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:43.146492 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ee8072-1e38-4713-baba-450ada01b394 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:43.150528 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed6d64c-570e-4a62-a469-8e29338d4127 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:43.151510 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed6d64c-570e-4a62-a469-8e29338d4127 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:43.152307 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed6d64c-570e-4a62-a469-8e29338d4127 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:43.152787 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed6d64c-570e-4a62-a469-8e29338d4127 None admin] PATH_INFO: `/v3/projects/cd69c40698884b3192bb19ba6968842d` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:43.156579 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed6d64c-570e-4a62-a469-8e29338d4127 None admin] RBAC: Authorizing `identity:get_project(project_id=cd69c40698884b3192bb19ba6968842d)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:43.158454 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed6d64c-570e-4a62-a469-8e29338d4127 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:43.161574 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 182/351] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:41:42 2022] GET /identity/v3/projects/cd69c40698884b3192bb19ba6968842d => generated 298 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:43.163079 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e34d1a4-46ce-4b00-a692-9d3154ae8fa6 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:43.163512 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e34d1a4-46ce-4b00-a692-9d3154ae8fa6 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:43.164380 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e34d1a4-46ce-4b00-a692-9d3154ae8fa6 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:43.166446 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:43.167784 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 183/352] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:43 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:43.169606 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e34d1a4-46ce-4b00-a692-9d3154ae8fa6 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:43.280966 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ee8072-1e38-4713-baba-450ada01b394 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:43.312041 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e34d1a4-46ce-4b00-a692-9d3154ae8fa6 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:43.348437 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e45f47-5a3a-4464-a2bd-5e1f1ef60738 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:43.349992 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e45f47-5a3a-4464-a2bd-5e1f1ef60738 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:43.350272 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e45f47-5a3a-4464-a2bd-5e1f1ef60738 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:43.350539 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e45f47-5a3a-4464-a2bd-5e1f1ef60738 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:43.352985 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e45f47-5a3a-4464-a2bd-5e1f1ef60738 None admin] RBAC: Adding query filter params (role.id=cc65bfe8ed2740b69427f705add475a5, user.id=66ba69ef18a14430b6b22fe756146274, scope.system=all) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:43.352985 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e45f47-5a3a-4464-a2bd-5e1f1ef60738 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:43.353588 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e45f47-5a3a-4464-a2bd-5e1f1ef60738 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:43.371176 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 170/353] 10.222.0.52 () {56 vars in 1308 bytes} [Sun Sep 25 00:41:43 2022] GET /identity/v3/role_assignments?user.id=66ba69ef18a14430b6b22fe756146274&scope.system=all&role.id=cc65bfe8ed2740b69427f705add475a5 => generated 510 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:43.373432 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e45f47-5a3a-4464-a2bd-5e1f1ef60738 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:43.379299 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e620e40-95b0-4a4f-93ae-cc76e757d76f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:43.380214 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e620e40-95b0-4a4f-93ae-cc76e757d76f None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:43.380691 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e620e40-95b0-4a4f-93ae-cc76e757d76f None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:43.381549 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e620e40-95b0-4a4f-93ae-cc76e757d76f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:43.466145 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:43.488913 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:43.511519 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e620e40-95b0-4a4f-93ae-cc76e757d76f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:43.514002 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e620e40-95b0-4a4f-93ae-cc76e757d76f None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:43.519711 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e45f47-5a3a-4464-a2bd-5e1f1ef60738 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:43.567356 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fa7e4c8-f35e-47df-8fd2-4d42bbd51000 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:43.568419 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa7e4c8-f35e-47df-8fd2-4d42bbd51000 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:43.568917 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa7e4c8-f35e-47df-8fd2-4d42bbd51000 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:43.569453 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa7e4c8-f35e-47df-8fd2-4d42bbd51000 None admin] PATH_INFO: `/v3/groups/f52544847fbb4d61b602bdeadf58d9d6` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:43.582174 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fa7e4c8-f35e-47df-8fd2-4d42bbd51000 None admin] RBAC: Authorizing `identity:get_group(group_id=f52544847fbb4d61b602bdeadf58d9d6)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:43.584052 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fa7e4c8-f35e-47df-8fd2-4d42bbd51000 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:43.586436 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 171/354] 10.222.0.52 () {56 vars in 1180 bytes} [Sun Sep 25 00:41:43 2022] GET /identity/v3/groups/f52544847fbb4d61b602bdeadf58d9d6 => generated 226 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:43.589665 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41252956-d35f-44e9-880a-5607d40af2f9 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:43.590227 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41252956-d35f-44e9-880a-5607d40af2f9 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:43.590703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41252956-d35f-44e9-880a-5607d40af2f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:43.611350 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-41252956-d35f-44e9-880a-5607d40af2f9 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:43.612568 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-41252956-d35f-44e9-880a-5607d40af2f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:43.666966 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 184/355] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:41:43 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 498 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 00:41:43.667234 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e620e40-95b0-4a4f-93ae-cc76e757d76f None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:43.725874 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 172/356] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:41:43.727095 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41252956-d35f-44e9-880a-5607d40af2f9 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:43.801684 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:43.806003 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:43.819309 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e620e40-95b0-4a4f-93ae-cc76e757d76f None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:43.849807 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41252956-d35f-44e9-880a-5607d40af2f9 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:43.877417 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f96b4fb-5dff-4ed8-b4d3-7ca1892f44e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:43.878371 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f96b4fb-5dff-4ed8-b4d3-7ca1892f44e5 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:43.878762 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f96b4fb-5dff-4ed8-b4d3-7ca1892f44e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:43.879126 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f96b4fb-5dff-4ed8-b4d3-7ca1892f44e5 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:43.889131 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83aaf983-3396-463d-baeb-9e14f767094a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:43.889774 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83aaf983-3396-463d-baeb-9e14f767094a None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:43.890046 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83aaf983-3396-463d-baeb-9e14f767094a None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:43.890309 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83aaf983-3396-463d-baeb-9e14f767094a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:43.891032 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83aaf983-3396-463d-baeb-9e14f767094a None admin] RBAC: Adding query filter params (group.id=f52544847fbb4d61b602bdeadf58d9d6, role.id=40f54f5ac0994c51960ac413da8d2cc9, scope.project.id=cd69c40698884b3192bb19ba6968842d) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:43.891330 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83aaf983-3396-463d-baeb-9e14f767094a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:43.892780 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83aaf983-3396-463d-baeb-9e14f767094a None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:43.892898 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f96b4fb-5dff-4ed8-b4d3-7ca1892f44e5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:41:43.897841 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-2f96b4fb-5dff-4ed8-b4d3-7ca1892f44e5 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 25 00:41:43.897841 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:41:43.897841 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:41:43.897841 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:41:43.897841 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:41:43.897841 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:41:43.897841 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:41:43.897841 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:41:43.897841 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:41:43.897841 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:41:43.897841 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:41:43.897841 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:41:43.897841 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:41:43.897841 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:41:43.897841 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:41:43.897841 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:41:43.898754 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:41:43.898754 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:41:43.898754 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:41:43.898754 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:41:43.898754 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:41:43.898754 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:41:43.898754 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:41:43.898754 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:41:43.898754 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:41:43.898754 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:41:43.898754 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:41:43.898754 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:41:43.898754 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:41:43.898754 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:41:43.898754 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:41:43.898754 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:41:43.900283 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:41:43.900283 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:41:43.900283 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:41:43.900283 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:41:43.900283 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:41:43.900283 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:41:43.900283 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:41:43.900283 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:41:43.900283 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:41:43.900283 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:41:43.900283 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 25 00:41:43.900283 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:41:43.900283 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 185/357] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:41:43 2022] GET /identity/v3/roles/member => generated 84 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:41:43.901153 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 173/358] 10.222.0.52 () {56 vars in 1376 bytes} [Sun Sep 25 00:41:43 2022] GET /identity/v3/role_assignments?group.id=f52544847fbb4d61b602bdeadf58d9d6&scope.project.id=cd69c40698884b3192bb19ba6968842d&role.id=40f54f5ac0994c51960ac413da8d2cc9 => generated 260 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:43.901233 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f96b4fb-5dff-4ed8-b4d3-7ca1892f44e5 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:43.922604 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83aaf983-3396-463d-baeb-9e14f767094a None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:44.105793 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83aaf983-3396-463d-baeb-9e14f767094a None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:44.118713 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f96b4fb-5dff-4ed8-b4d3-7ca1892f44e5 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:44.142484 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:44.156026 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c7cb4c-1924-4fda-bd94-83ace82f3dc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:44.156647 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c7cb4c-1924-4fda-bd94-83ace82f3dc2 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:44.156925 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c7cb4c-1924-4fda-bd94-83ace82f3dc2 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:44.157180 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c7cb4c-1924-4fda-bd94-83ace82f3dc2 None admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:44.158122 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94c7cb4c-1924-4fda-bd94-83ace82f3dc2 None admin] RBAC: Adding query filter params (name=member) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:44.158394 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94c7cb4c-1924-4fda-bd94-83ace82f3dc2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:44.160203 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94c7cb4c-1924-4fda-bd94-83ace82f3dc2 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:44.170012 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d785b15f-7498-4803-a33a-e88a0eca9de1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:44.170698 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d785b15f-7498-4803-a33a-e88a0eca9de1 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:44.170917 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d785b15f-7498-4803-a33a-e88a0eca9de1 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:44.171186 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d785b15f-7498-4803-a33a-e88a0eca9de1 None admin] PATH_INFO: `/v3/projects/cd69c40698884b3192bb19ba6968842d` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:44.171290 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 174/359] 10.222.0.52 () {56 vars in 1101 bytes} [Sun Sep 25 00:41:43 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:44.172082 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c7cb4c-1924-4fda-bd94-83ace82f3dc2 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:44.174071 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d785b15f-7498-4803-a33a-e88a0eca9de1 None admin] RBAC: Authorizing `identity:get_project(project_id=cd69c40698884b3192bb19ba6968842d)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:44.175881 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d785b15f-7498-4803-a33a-e88a0eca9de1 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:44.181288 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 186/360] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:41:43 2022] GET /identity/v3/projects/cd69c40698884b3192bb19ba6968842d => generated 298 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:44.181288 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4b0ce3-0eeb-4f5c-8ca4-6e5572754d52 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:44.181288 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4b0ce3-0eeb-4f5c-8ca4-6e5572754d52 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:44.181288 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4b0ce3-0eeb-4f5c-8ca4-6e5572754d52 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:44.181288 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 187/361] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:44 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:44.181696 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c4b0ce3-0eeb-4f5c-8ca4-6e5572754d52 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:44.209347 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:44.316679 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c7cb4c-1924-4fda-bd94-83ace82f3dc2 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:44.322142 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c4b0ce3-0eeb-4f5c-8ca4-6e5572754d52 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:44.374106 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ddc6c3a-451c-4ef8-89a5-c46882042715 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:44.374723 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ddc6c3a-451c-4ef8-89a5-c46882042715 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:44.374995 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ddc6c3a-451c-4ef8-89a5-c46882042715 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:44.375258 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ddc6c3a-451c-4ef8-89a5-c46882042715 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:44.377970 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db4db9e-fc4e-4253-9613-35847234d419 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:44.378603 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db4db9e-fc4e-4253-9613-35847234d419 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:44.378877 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db4db9e-fc4e-4253-9613-35847234d419 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:44.379143 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db4db9e-fc4e-4253-9613-35847234d419 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:44.396193 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db4db9e-fc4e-4253-9613-35847234d419 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:41:44.402937 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-9db4db9e-fc4e-4253-9613-35847234d419 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 00:41:44.402937 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:41:44.402937 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:41:44.402937 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:41:44.402937 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:41:44.402937 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:41:44.402937 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:41:44.402937 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:41:44.402937 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:41:44.402937 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:41:44.402937 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:41:44.402937 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:41:44.402937 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:41:44.402937 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:41:44.402937 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:41:44.402937 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:41:44.403890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:41:44.403890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:41:44.403890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:41:44.403890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:41:44.403890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:41:44.403890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:41:44.403890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:41:44.403890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:41:44.403890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:41:44.403890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:41:44.403890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:41:44.403890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:41:44.403890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:41:44.403890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:41:44.403890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:41:44.403890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:41:44.404715 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:41:44.404715 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:41:44.404715 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:41:44.404715 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:41:44.404715 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:41:44.404715 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:41:44.404715 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:41:44.404715 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:41:44.404715 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:41:44.404715 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:41:44.404715 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 00:41:44.404715 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:41:44.404715 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 175/362] 10.222.0.52 () {56 vars in 1114 bytes} [Sun Sep 25 00:41:44 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:41:44.406712 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db4db9e-fc4e-4253-9613-35847234d419 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:44.466297 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ddc6c3a-451c-4ef8-89a5-c46882042715 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:44.467656 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ddc6c3a-451c-4ef8-89a5-c46882042715 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:44.468396 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:44.507325 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:44.546135 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db4db9e-fc4e-4253-9613-35847234d419 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:44.596369 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6210ce3-580e-4c1a-85b0-a6b0cae705e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:44.597104 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6210ce3-580e-4c1a-85b0-a6b0cae705e8 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:44.597421 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6210ce3-580e-4c1a-85b0-a6b0cae705e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:44.597701 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6210ce3-580e-4c1a-85b0-a6b0cae705e8 None admin] PATH_INFO: `/v3/groups/f52544847fbb4d61b602bdeadf58d9d6` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:44.598845 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6210ce3-580e-4c1a-85b0-a6b0cae705e8 None admin] RBAC: Authorizing `identity:get_group(group_id=f52544847fbb4d61b602bdeadf58d9d6)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:44.600318 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6210ce3-580e-4c1a-85b0-a6b0cae705e8 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:44.601948 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 176/363] 10.222.0.52 () {56 vars in 1180 bytes} [Sun Sep 25 00:41:44 2022] GET /identity/v3/groups/f52544847fbb4d61b602bdeadf58d9d6 => generated 226 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:44.603164 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b3d01a4-1fae-464d-8ad6-0327ee614408 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:44.603442 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b3d01a4-1fae-464d-8ad6-0327ee614408 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:44.603712 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b3d01a4-1fae-464d-8ad6-0327ee614408 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:44.627039 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-4b3d01a4-1fae-464d-8ad6-0327ee614408 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:44.628582 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-4b3d01a4-1fae-464d-8ad6-0327ee614408 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:44.631640 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 188/364] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:41:44 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 450 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 00:41:44.632299 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ddc6c3a-451c-4ef8-89a5-c46882042715 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:44.727203 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 177/365] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:41:44.728395 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b3d01a4-1fae-464d-8ad6-0327ee614408 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:44.764555 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ddc6c3a-451c-4ef8-89a5-c46882042715 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:44.791630 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:44.794752 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:44.813869 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4a2dea-626d-4618-9eda-878568a13eb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:44.814718 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4a2dea-626d-4618-9eda-878568a13eb3 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:44.815169 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4a2dea-626d-4618-9eda-878568a13eb3 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:44.815579 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4a2dea-626d-4618-9eda-878568a13eb3 None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:44.816390 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c4a2dea-626d-4618-9eda-878568a13eb3 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:44.816833 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c4a2dea-626d-4618-9eda-878568a13eb3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:44.818696 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c4a2dea-626d-4618-9eda-878568a13eb3 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:44.829916 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 189/366] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:41:44 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:44.831134 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4a2dea-626d-4618-9eda-878568a13eb3 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:44.865251 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b3d01a4-1fae-464d-8ad6-0327ee614408 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:44.922495 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ab96c9-50e0-4110-ba1f-3c9f9fca34ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:44.923116 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ab96c9-50e0-4110-ba1f-3c9f9fca34ca None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:44.923390 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ab96c9-50e0-4110-ba1f-3c9f9fca34ca None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:44.923657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ab96c9-50e0-4110-ba1f-3c9f9fca34ca None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:44.924388 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2ab96c9-50e0-4110-ba1f-3c9f9fca34ca None admin] RBAC: Adding query filter params (group.id=f52544847fbb4d61b602bdeadf58d9d6, role.id=a917909d7aa84d00a841c77e7e7bdc8a, scope.project.id=cd69c40698884b3192bb19ba6968842d) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:44.924689 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2ab96c9-50e0-4110-ba1f-3c9f9fca34ca None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:44.928721 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2ab96c9-50e0-4110-ba1f-3c9f9fca34ca None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:44.935671 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 178/367] 10.222.0.52 () {56 vars in 1376 bytes} [Sun Sep 25 00:41:44 2022] GET /identity/v3/role_assignments?group.id=f52544847fbb4d61b602bdeadf58d9d6&scope.project.id=cd69c40698884b3192bb19ba6968842d&role.id=a917909d7aa84d00a841c77e7e7bdc8a => generated 260 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:44.936673 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ab96c9-50e0-4110-ba1f-3c9f9fca34ca None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:44.953521 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4a2dea-626d-4618-9eda-878568a13eb3 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:45.006336 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-209a487a-13ab-4244-a432-fd62e6c34b10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:45.006884 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-209a487a-13ab-4244-a432-fd62e6c34b10 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:45.007154 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-209a487a-13ab-4244-a432-fd62e6c34b10 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:45.007463 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-209a487a-13ab-4244-a432-fd62e6c34b10 None admin] PATH_INFO: `/v3/projects/86aea44da5fb4bdeb527c7ff257c5d43` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:45.009271 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-209a487a-13ab-4244-a432-fd62e6c34b10 None admin] RBAC: Authorizing `identity:get_project(project_id=86aea44da5fb4bdeb527c7ff257c5d43)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:45.011135 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-209a487a-13ab-4244-a432-fd62e6c34b10 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:45.012860 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 190/368] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:41:44 2022] GET /identity/v3/projects/86aea44da5fb4bdeb527c7ff257c5d43 => generated 302 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:45.019408 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-209a487a-13ab-4244-a432-fd62e6c34b10 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:45.075953 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ab96c9-50e0-4110-ba1f-3c9f9fca34ca None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:45.086194 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:45.094372 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:45.125079 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a71a050b-64a8-4d5f-a584-94e36a65cba4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:45.125743 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71a050b-64a8-4d5f-a584-94e36a65cba4 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:45.126080 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71a050b-64a8-4d5f-a584-94e36a65cba4 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:45.126355 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71a050b-64a8-4d5f-a584-94e36a65cba4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:45.162473 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-209a487a-13ab-4244-a432-fd62e6c34b10 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:45.218943 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a71a050b-64a8-4d5f-a584-94e36a65cba4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:45.219879 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85eace42-3ede-499c-b525-4984a7cfa38d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:45.220532 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85eace42-3ede-499c-b525-4984a7cfa38d None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:45.220851 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85eace42-3ede-499c-b525-4984a7cfa38d None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:45.221122 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85eace42-3ede-499c-b525-4984a7cfa38d None admin] PATH_INFO: `/v3/groups/f52544847fbb4d61b602bdeadf58d9d6` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:45.222303 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a71a050b-64a8-4d5f-a584-94e36a65cba4 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:45.222440 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85eace42-3ede-499c-b525-4984a7cfa38d None admin] RBAC: Authorizing `identity:get_group(group_id=f52544847fbb4d61b602bdeadf58d9d6)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:45.223871 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85eace42-3ede-499c-b525-4984a7cfa38d None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:45.229284 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 191/369] 10.222.0.52 () {56 vars in 1180 bytes} [Sun Sep 25 00:41:45 2022] GET /identity/v3/groups/f52544847fbb4d61b602bdeadf58d9d6 => generated 226 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:45.229284 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85eace42-3ede-499c-b525-4984a7cfa38d None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:45.366422 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85eace42-3ede-499c-b525-4984a7cfa38d None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:45.387928 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 179/370] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:41:44 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 452 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 00:41:45.390981 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a71a050b-64a8-4d5f-a584-94e36a65cba4 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:45.400967 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:45.411763 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:45.423038 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40578d38-1949-4818-b783-bbd8e932a936 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:45.424054 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40578d38-1949-4818-b783-bbd8e932a936 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:45.424556 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40578d38-1949-4818-b783-bbd8e932a936 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:45.425039 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40578d38-1949-4818-b783-bbd8e932a936 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:45.439609 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40578d38-1949-4818-b783-bbd8e932a936 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:41:45.445656 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-40578d38-1949-4818-b783-bbd8e932a936 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:41:45.445656 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:41:45.445656 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:41:45.445656 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:41:45.445656 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:41:45.445656 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:41:45.445656 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:41:45.445656 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:41:45.445656 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:41:45.445656 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:41:45.445656 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:41:45.445656 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:41:45.445656 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:41:45.445656 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:41:45.445656 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:41:45.445656 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:41:45.447334 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:41:45.447334 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:41:45.447334 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:41:45.447334 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:41:45.447334 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:41:45.447334 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:41:45.447334 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:41:45.447334 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:41:45.447334 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:41:45.447334 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:41:45.447334 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:41:45.447334 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:41:45.447334 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:41:45.447334 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:41:45.447334 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:41:45.447334 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:41:45.448172 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:41:45.448172 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:41:45.448172 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:41:45.448172 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:41:45.448172 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:41:45.448172 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:41:45.448172 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:41:45.448172 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:41:45.448172 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:41:45.448172 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:41:45.448172 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:41:45.448172 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:41:45.451599 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 192/371] 10.222.0.52 () {56 vars in 1096 bytes} [Sun Sep 25 00:41:45 2022] GET /identity/v3/roles/admin => generated 83 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:41:45.452714 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40578d38-1949-4818-b783-bbd8e932a936 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:45.537572 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a71a050b-64a8-4d5f-a584-94e36a65cba4 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:45.578894 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f0b9e0f-5bf2-46ce-8512-794fda2f4db1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:45.579600 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0b9e0f-5bf2-46ce-8512-794fda2f4db1 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:45.579948 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0b9e0f-5bf2-46ce-8512-794fda2f4db1 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:45.580287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0b9e0f-5bf2-46ce-8512-794fda2f4db1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:45.581081 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f0b9e0f-5bf2-46ce-8512-794fda2f4db1 None admin] RBAC: Adding query filter params (group.id=f52544847fbb4d61b602bdeadf58d9d6, role.id=40f54f5ac0994c51960ac413da8d2cc9, scope.project.id=86aea44da5fb4bdeb527c7ff257c5d43) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:45.581506 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f0b9e0f-5bf2-46ce-8512-794fda2f4db1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:45.583211 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f0b9e0f-5bf2-46ce-8512-794fda2f4db1 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:45.592046 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 180/372] 10.222.0.52 () {56 vars in 1376 bytes} [Sun Sep 25 00:41:45 2022] GET /identity/v3/role_assignments?group.id=f52544847fbb4d61b602bdeadf58d9d6&scope.project.id=86aea44da5fb4bdeb527c7ff257c5d43&role.id=40f54f5ac0994c51960ac413da8d2cc9 => generated 260 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:45.592046 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f0b9e0f-5bf2-46ce-8512-794fda2f4db1 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:45.596544 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40578d38-1949-4818-b783-bbd8e932a936 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:45.643994 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa1be49-e6b3-494c-aadf-b1b6aa520f1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:45.645545 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa1be49-e6b3-494c-aadf-b1b6aa520f1e None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:45.646275 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa1be49-e6b3-494c-aadf-b1b6aa520f1e None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:45.646701 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa1be49-e6b3-494c-aadf-b1b6aa520f1e None admin] PATH_INFO: `/v3/projects/86aea44da5fb4bdeb527c7ff257c5d43` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:45.648330 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aa1be49-e6b3-494c-aadf-b1b6aa520f1e None admin] RBAC: Authorizing `identity:get_project(project_id=86aea44da5fb4bdeb527c7ff257c5d43)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:45.650179 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aa1be49-e6b3-494c-aadf-b1b6aa520f1e None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:45.652123 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 193/373] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:41:45 2022] GET /identity/v3/projects/86aea44da5fb4bdeb527c7ff257c5d43 => generated 302 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:45.659981 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa1be49-e6b3-494c-aadf-b1b6aa520f1e None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:45.690035 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:45.732439 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f0b9e0f-5bf2-46ce-8512-794fda2f4db1 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:45.737615 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:45.810093 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d61d8bc-2027-4a22-be42-b76c9f7017ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:45.810835 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d61d8bc-2027-4a22-be42-b76c9f7017ff None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:45.811190 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d61d8bc-2027-4a22-be42-b76c9f7017ff None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:45.811527 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d61d8bc-2027-4a22-be42-b76c9f7017ff None admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:45.812295 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d61d8bc-2027-4a22-be42-b76c9f7017ff None admin] RBAC: Adding query filter params (name=admin) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:45.812676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d61d8bc-2027-4a22-be42-b76c9f7017ff None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:45.814158 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d61d8bc-2027-4a22-be42-b76c9f7017ff None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:45.828938 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 181/374] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:41:45 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:45.832326 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa1be49-e6b3-494c-aadf-b1b6aa520f1e None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:45.834760 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d61d8bc-2027-4a22-be42-b76c9f7017ff None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:45.892040 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb74ec96-a559-480e-bce9-37ea259ed82e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:45.892691 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb74ec96-a559-480e-bce9-37ea259ed82e None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:45.892966 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb74ec96-a559-480e-bce9-37ea259ed82e None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:45.893233 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb74ec96-a559-480e-bce9-37ea259ed82e None admin] PATH_INFO: `/v3/groups/f52544847fbb4d61b602bdeadf58d9d6` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:45.895258 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb74ec96-a559-480e-bce9-37ea259ed82e None admin] RBAC: Authorizing `identity:get_group(group_id=f52544847fbb4d61b602bdeadf58d9d6)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:45.896667 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb74ec96-a559-480e-bce9-37ea259ed82e None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:45.898548 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 194/375] 10.222.0.52 () {56 vars in 1180 bytes} [Sun Sep 25 00:41:45 2022] GET /identity/v3/groups/f52544847fbb4d61b602bdeadf58d9d6 => generated 226 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:41:45.910896 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb74ec96-a559-480e-bce9-37ea259ed82e None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:46.033122 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d61d8bc-2027-4a22-be42-b76c9f7017ff None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:46.090807 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b59148-1513-4d43-8f62-fb0cb0de32f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:46.091431 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b59148-1513-4d43-8f62-fb0cb0de32f3 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:46.091709 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b59148-1513-4d43-8f62-fb0cb0de32f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:46.091983 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b59148-1513-4d43-8f62-fb0cb0de32f3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:46.098122 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:46.113084 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb74ec96-a559-480e-bce9-37ea259ed82e None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:46.131762 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:46.175254 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec83b1cf-e8de-470f-a03c-f56e632f8829 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:46.175791 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec83b1cf-e8de-470f-a03c-f56e632f8829 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:46.176066 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec83b1cf-e8de-470f-a03c-f56e632f8829 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:46.176331 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec83b1cf-e8de-470f-a03c-f56e632f8829 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:46.177059 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec83b1cf-e8de-470f-a03c-f56e632f8829 None admin] RBAC: Adding query filter params (group.id=f52544847fbb4d61b602bdeadf58d9d6, role.id=a917909d7aa84d00a841c77e7e7bdc8a, scope.project.id=86aea44da5fb4bdeb527c7ff257c5d43) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:46.177387 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec83b1cf-e8de-470f-a03c-f56e632f8829 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:46.178825 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec83b1cf-e8de-470f-a03c-f56e632f8829 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:46.189145 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 195/376] 10.222.0.52 () {56 vars in 1376 bytes} [Sun Sep 25 00:41:45 2022] GET /identity/v3/role_assignments?group.id=f52544847fbb4d61b602bdeadf58d9d6&scope.project.id=86aea44da5fb4bdeb527c7ff257c5d43&role.id=a917909d7aa84d00a841c77e7e7bdc8a => generated 260 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:41:46.215018 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6b59148-1513-4d43-8f62-fb0cb0de32f3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:46.216516 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6b59148-1513-4d43-8f62-fb0cb0de32f3 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:46.347818 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 182/377] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:41:45 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 513 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 25 00:41:46.353544 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec83b1cf-e8de-470f-a03c-f56e632f8829 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:46.482668 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec83b1cf-e8de-470f-a03c-f56e632f8829 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:46.528324 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09dd8c42-8b26-4e48-b969-b280508f9a89 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:46.528650 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09dd8c42-8b26-4e48-b969-b280508f9a89 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:46.528870 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09dd8c42-8b26-4e48-b969-b280508f9a89 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:46.530246 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 183/378] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 00:41:46.531989 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc07602-d64c-4b0a-a445-d7e0dcbba29f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:46.532650 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc07602-d64c-4b0a-a445-d7e0dcbba29f None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:46.532972 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc07602-d64c-4b0a-a445-d7e0dcbba29f None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:46.533245 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc07602-d64c-4b0a-a445-d7e0dcbba29f None admin] PATH_INFO: `/v3/projects/c8e32a9668254c2e96354fd797883364` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:46.534470 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dc07602-d64c-4b0a-a445-d7e0dcbba29f None admin] RBAC: Authorizing `identity:get_project(project_id=c8e32a9668254c2e96354fd797883364)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:46.535919 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dc07602-d64c-4b0a-a445-d7e0dcbba29f None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:46.538557 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a611a296-26ba-46f7-82e6-3158cf265d61 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:46.538838 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a611a296-26ba-46f7-82e6-3158cf265d61 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:46.539101 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a611a296-26ba-46f7-82e6-3158cf265d61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:46.540045 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 196/379] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:41:46 2022] GET /identity/v3/projects/c8e32a9668254c2e96354fd797883364 => generated 344 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:46.545489 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc07602-d64c-4b0a-a445-d7e0dcbba29f None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:46.580537 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-a611a296-26ba-46f7-82e6-3158cf265d61 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:46.581801 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-a611a296-26ba-46f7-82e6-3158cf265d61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:46.640834 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:46.650706 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:46.735486 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 184/380] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 363 bytes (3 switches on core 0) Sep 25 00:41:46.740796 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc07602-d64c-4b0a-a445-d7e0dcbba29f None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:46.779823 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49363651-1128-4537-a91a-3eea56b8747a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:46.780461 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49363651-1128-4537-a91a-3eea56b8747a None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:46.780734 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49363651-1128-4537-a91a-3eea56b8747a None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:46.780999 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49363651-1128-4537-a91a-3eea56b8747a None admin] PATH_INFO: `/v3/groups/3593b329027d4ed7abe6cc428574877a` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:46.790844 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49363651-1128-4537-a91a-3eea56b8747a None admin] RBAC: Authorizing `identity:get_group(group_id=3593b329027d4ed7abe6cc428574877a)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:46.792385 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49363651-1128-4537-a91a-3eea56b8747a None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:46.794352 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 197/381] 10.222.0.52 () {56 vars in 1180 bytes} [Sun Sep 25 00:41:46 2022] GET /identity/v3/groups/3593b329027d4ed7abe6cc428574877a => generated 229 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:46.800006 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a611a296-26ba-46f7-82e6-3158cf265d61 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:46.926435 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a611a296-26ba-46f7-82e6-3158cf265d61 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:46.973907 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5abf6e1f-efa5-49a5-939d-12100f388540 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:46.974517 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5abf6e1f-efa5-49a5-939d-12100f388540 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:46.974794 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5abf6e1f-efa5-49a5-939d-12100f388540 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:46.975062 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5abf6e1f-efa5-49a5-939d-12100f388540 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:46.975787 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5abf6e1f-efa5-49a5-939d-12100f388540 None admin] RBAC: Adding query filter params (group.id=3593b329027d4ed7abe6cc428574877a, role.id=2c242abf6ed8410ab0a4a354ca1794fb, scope.project.id=c8e32a9668254c2e96354fd797883364) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:46.976092 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5abf6e1f-efa5-49a5-939d-12100f388540 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:46.978063 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5abf6e1f-efa5-49a5-939d-12100f388540 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:46.983926 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 185/382] 10.222.0.52 () {56 vars in 1376 bytes} [Sun Sep 25 00:41:46 2022] GET /identity/v3/role_assignments?group.id=3593b329027d4ed7abe6cc428574877a&scope.project.id=c8e32a9668254c2e96354fd797883364&role.id=2c242abf6ed8410ab0a4a354ca1794fb => generated 260 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:47.080872 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49363651-1128-4537-a91a-3eea56b8747a None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:47.222065 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49363651-1128-4537-a91a-3eea56b8747a None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:47.235947 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:47.263843 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b211a61e-0e4a-47c0-b78e-88dc1186e2b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:47.264812 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b211a61e-0e4a-47c0-b78e-88dc1186e2b8 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:47.265370 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b211a61e-0e4a-47c0-b78e-88dc1186e2b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:47.265829 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b211a61e-0e4a-47c0-b78e-88dc1186e2b8 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:47.277090 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b211a61e-0e4a-47c0-b78e-88dc1186e2b8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:41:47.282794 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-b211a61e-0e4a-47c0-b78e-88dc1186e2b8 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 25 00:41:47.282794 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:41:47.282794 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:41:47.282794 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:41:47.282794 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:41:47.282794 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:41:47.282794 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:41:47.282794 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:41:47.282794 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:41:47.282794 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:41:47.282794 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:41:47.282794 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:41:47.282794 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:41:47.282794 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:41:47.282794 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:41:47.282794 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:41:47.283713 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:41:47.283713 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:41:47.283713 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:41:47.283713 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:41:47.283713 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:41:47.283713 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:41:47.283713 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:41:47.283713 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:41:47.283713 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:41:47.283713 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:41:47.283713 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:41:47.283713 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:41:47.283713 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:41:47.283713 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:41:47.283713 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:41:47.283713 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:41:47.284527 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:41:47.284527 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:41:47.284527 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:41:47.284527 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:41:47.284527 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:41:47.284527 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:41:47.284527 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:41:47.284527 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:41:47.284527 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:41:47.284527 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:41:47.284527 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 25 00:41:47.284527 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:41:47.286832 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 198/383] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:41:47 2022] GET /identity/v3/roles/member => generated 84 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:41:47.292754 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5abf6e1f-efa5-49a5-939d-12100f388540 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:47.332617 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:47.422336 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5abf6e1f-efa5-49a5-939d-12100f388540 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:47.472626 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b71f0d-0630-4475-84df-614b2835d184 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:47.473691 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b71f0d-0630-4475-84df-614b2835d184 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:47.474177 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b71f0d-0630-4475-84df-614b2835d184 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:47.474639 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b71f0d-0630-4475-84df-614b2835d184 None admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:47.475505 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b71f0d-0630-4475-84df-614b2835d184 None admin] RBAC: Adding query filter params (name=member) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:47.476007 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b71f0d-0630-4475-84df-614b2835d184 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:47.478049 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b71f0d-0630-4475-84df-614b2835d184 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:47.490748 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 186/384] 10.222.0.52 () {56 vars in 1101 bytes} [Sun Sep 25 00:41:47 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:41:47.498940 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b211a61e-0e4a-47c0-b78e-88dc1186e2b8 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:47.656261 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b211a61e-0e4a-47c0-b78e-88dc1186e2b8 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:47.701606 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4322b70-4a45-4939-ac34-4fbf64d118f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:47.702257 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4322b70-4a45-4939-ac34-4fbf64d118f0 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:47.702529 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4322b70-4a45-4939-ac34-4fbf64d118f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:47.702796 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4322b70-4a45-4939-ac34-4fbf64d118f0 None admin] PATH_INFO: `/v3/groups/f52544847fbb4d61b602bdeadf58d9d6` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:47.703932 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4322b70-4a45-4939-ac34-4fbf64d118f0 None admin] RBAC: Authorizing `identity:get_group(group_id=f52544847fbb4d61b602bdeadf58d9d6)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:47.705400 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4322b70-4a45-4939-ac34-4fbf64d118f0 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:47.707184 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 199/385] 10.222.0.52 () {56 vars in 1180 bytes} [Sun Sep 25 00:41:47 2022] GET /identity/v3/groups/f52544847fbb4d61b602bdeadf58d9d6 => generated 226 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:47.713562 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b71f0d-0630-4475-84df-614b2835d184 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:47.843802 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:47.854444 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b71f0d-0630-4475-84df-614b2835d184 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:47.898435 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d251aec3-2289-4782-87c8-4e51ae89a107 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:47.899126 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d251aec3-2289-4782-87c8-4e51ae89a107 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:47.899404 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d251aec3-2289-4782-87c8-4e51ae89a107 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:47.899673 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d251aec3-2289-4782-87c8-4e51ae89a107 None admin] PATH_INFO: `/v3/projects/cd69c40698884b3192bb19ba6968842d` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:47.900993 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d251aec3-2289-4782-87c8-4e51ae89a107 None admin] RBAC: Authorizing `identity:get_project(project_id=cd69c40698884b3192bb19ba6968842d)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:47.902469 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d251aec3-2289-4782-87c8-4e51ae89a107 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:47.904477 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 187/386] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:41:47 2022] GET /identity/v3/projects/cd69c40698884b3192bb19ba6968842d => generated 298 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:47.910353 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4322b70-4a45-4939-ac34-4fbf64d118f0 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:47.990969 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:48.081432 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4322b70-4a45-4939-ac34-4fbf64d118f0 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:48.127166 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf6023c-d6db-41fc-9b8c-d2adb70e7123 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:48.127833 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf6023c-d6db-41fc-9b8c-d2adb70e7123 None admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:48.128108 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf6023c-d6db-41fc-9b8c-d2adb70e7123 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:48.128376 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf6023c-d6db-41fc-9b8c-d2adb70e7123 None admin] PATH_INFO: `/v3/projects/cd69c40698884b3192bb19ba6968842d/groups/f52544847fbb4d61b602bdeadf58d9d6/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:48.131517 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cf6023c-d6db-41fc-9b8c-d2adb70e7123 None admin] RBAC: Authorizing `identity:create_grant(project_id=cd69c40698884b3192bb19ba6968842d, group_id=f52544847fbb4d61b602bdeadf58d9d6, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:48.132969 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cf6023c-d6db-41fc-9b8c-d2adb70e7123 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:48.157644 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 200/387] 10.222.0.52 () {58 vars in 1442 bytes} [Sun Sep 25 00:41:47 2022] PUT /identity/v3/projects/cd69c40698884b3192bb19ba6968842d/groups/f52544847fbb4d61b602bdeadf58d9d6/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:41:48.157862 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6289e3db-eefe-4d36-bfdb-3fd74e1259d0 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:48.158382 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6289e3db-eefe-4d36-bfdb-3fd74e1259d0 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:48.158863 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6289e3db-eefe-4d36-bfdb-3fd74e1259d0 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:48.160742 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 188/388] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:48.166857 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5545f8b-e6e5-4d0b-912b-12da19f3f5c8 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:48.167172 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5545f8b-e6e5-4d0b-912b-12da19f3f5c8 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:48.167402 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5545f8b-e6e5-4d0b-912b-12da19f3f5c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:48.188534 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-c5545f8b-e6e5-4d0b-912b-12da19f3f5c8 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:48.189801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-c5545f8b-e6e5-4d0b-912b-12da19f3f5c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:48.274750 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 201/389] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:41:48.558986 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845fb6bb-15a0-4de9-a107-a4ff0d7a09d8 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:48.559948 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845fb6bb-15a0-4de9-a107-a4ff0d7a09d8 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:48.560440 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845fb6bb-15a0-4de9-a107-a4ff0d7a09d8 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:48.562262 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 189/390] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:48.567804 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04abe56-a3f5-4268-b63b-f9ba22e7c20b None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:48.568678 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04abe56-a3f5-4268-b63b-f9ba22e7c20b None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:48.569143 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04abe56-a3f5-4268-b63b-f9ba22e7c20b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:48.593748 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-e04abe56-a3f5-4268-b63b-f9ba22e7c20b None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:48.595307 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-e04abe56-a3f5-4268-b63b-f9ba22e7c20b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:48.623130 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845fb6bb-15a0-4de9-a107-a4ff0d7a09d8 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:48.628155 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:48.715387 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 202/391] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:41:48.781156 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845fb6bb-15a0-4de9-a107-a4ff0d7a09d8 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:48.819161 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b865fa68-9140-45ca-9009-ab3a15366cb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:48.820072 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b865fa68-9140-45ca-9009-ab3a15366cb9 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:48.820494 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b865fa68-9140-45ca-9009-ab3a15366cb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:48.820985 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b865fa68-9140-45ca-9009-ab3a15366cb9 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:48.832997 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b865fa68-9140-45ca-9009-ab3a15366cb9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:41:48.839967 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-b865fa68-9140-45ca-9009-ab3a15366cb9 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 00:41:48.839967 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:41:48.839967 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:41:48.839967 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:41:48.839967 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:41:48.839967 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:41:48.839967 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:41:48.839967 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:41:48.839967 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:41:48.839967 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:41:48.839967 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:41:48.839967 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:41:48.839967 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:41:48.839967 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:41:48.839967 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:41:48.839967 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:41:48.841079 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:41:48.841079 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:41:48.841079 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:41:48.841079 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:41:48.841079 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:41:48.841079 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:41:48.841079 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:41:48.841079 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:41:48.841079 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:41:48.841079 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:41:48.841079 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:41:48.841079 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:41:48.841079 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:41:48.841079 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:41:48.841079 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:41:48.841079 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:41:48.842145 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:41:48.842145 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:41:48.842145 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:41:48.842145 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:41:48.842145 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:41:48.842145 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:41:48.842145 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:41:48.842145 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:41:48.842145 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:41:48.842145 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:41:48.842145 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 00:41:48.842145 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:41:48.844460 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 190/392] 10.222.0.52 () {56 vars in 1114 bytes} [Sun Sep 25 00:41:48 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:41:48.849974 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e04abe56-a3f5-4268-b63b-f9ba22e7c20b None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:48.961998 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e04abe56-a3f5-4268-b63b-f9ba22e7c20b None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:49.013625 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6e2ce6-bffa-41f0-a283-83f60df753bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:49.014556 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6e2ce6-bffa-41f0-a283-83f60df753bc None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:49.014973 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6e2ce6-bffa-41f0-a283-83f60df753bc None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:49.015391 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6e2ce6-bffa-41f0-a283-83f60df753bc None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:49.016206 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b6e2ce6-bffa-41f0-a283-83f60df753bc None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:49.016648 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b6e2ce6-bffa-41f0-a283-83f60df753bc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:49.018190 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b6e2ce6-bffa-41f0-a283-83f60df753bc None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:49.031688 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 203/393] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:41:48 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:49.036991 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b865fa68-9140-45ca-9009-ab3a15366cb9 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:49.063596 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6e2ce6-bffa-41f0-a283-83f60df753bc None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:49.079629 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:49.091416 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:49.230118 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b865fa68-9140-45ca-9009-ab3a15366cb9 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:49.244155 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6e2ce6-bffa-41f0-a283-83f60df753bc None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:49.295893 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdbaf3cf-2eb5-42f4-9970-c584b3511950 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:49.296444 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdbaf3cf-2eb5-42f4-9970-c584b3511950 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:49.296718 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdbaf3cf-2eb5-42f4-9970-c584b3511950 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:49.296988 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdbaf3cf-2eb5-42f4-9970-c584b3511950 None admin] PATH_INFO: `/v3/groups/f52544847fbb4d61b602bdeadf58d9d6` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:49.298256 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdbaf3cf-2eb5-42f4-9970-c584b3511950 None admin] RBAC: Authorizing `identity:get_group(group_id=f52544847fbb4d61b602bdeadf58d9d6)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:49.299702 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdbaf3cf-2eb5-42f4-9970-c584b3511950 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:49.301507 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 191/394] 10.222.0.52 () {56 vars in 1180 bytes} [Sun Sep 25 00:41:49 2022] GET /identity/v3/groups/f52544847fbb4d61b602bdeadf58d9d6 => generated 226 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:49.309519 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdbaf3cf-2eb5-42f4-9970-c584b3511950 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:49.310915 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c8beaf1-9d00-4d1a-a3e8-521d1839540d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:49.311567 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8beaf1-9d00-4d1a-a3e8-521d1839540d None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:49.311841 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8beaf1-9d00-4d1a-a3e8-521d1839540d None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:49.312102 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8beaf1-9d00-4d1a-a3e8-521d1839540d None admin] PATH_INFO: `/v3/roles/member` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:49.335842 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c8beaf1-9d00-4d1a-a3e8-521d1839540d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:41:49.341546 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-4c8beaf1-9d00-4d1a-a3e8-521d1839540d None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 25 00:41:49.341546 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:41:49.341546 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:41:49.341546 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:41:49.341546 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:41:49.341546 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:41:49.341546 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:41:49.341546 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:41:49.341546 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:41:49.341546 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:41:49.341546 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:41:49.341546 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:41:49.341546 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:41:49.341546 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:41:49.341546 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:41:49.341546 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:41:49.343767 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:41:49.343767 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:41:49.343767 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:41:49.343767 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:41:49.343767 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:41:49.343767 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:41:49.343767 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:41:49.343767 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:41:49.343767 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:41:49.343767 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:41:49.343767 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:41:49.343767 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:41:49.343767 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:41:49.343767 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:41:49.343767 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:41:49.343767 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:41:49.353845 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:41:49.353845 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:41:49.353845 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:41:49.353845 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:41:49.353845 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:41:49.353845 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:41:49.353845 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:41:49.353845 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:41:49.353845 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:41:49.353845 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:41:49.353845 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 25 00:41:49.353845 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:41:49.353845 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 204/395] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:41:49 2022] GET /identity/v3/roles/member => generated 84 bytes in 280 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:41:49.355457 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c8beaf1-9d00-4d1a-a3e8-521d1839540d None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:49.510190 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:49.528033 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdbaf3cf-2eb5-42f4-9970-c584b3511950 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:49.542255 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:49.577242 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c8beaf1-9d00-4d1a-a3e8-521d1839540d None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:49.601388 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d49e6d-4ee4-432d-a354-710237f4e9b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:49.602053 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d49e6d-4ee4-432d-a354-710237f4e9b6 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:49.602517 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d49e6d-4ee4-432d-a354-710237f4e9b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:49.602805 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d49e6d-4ee4-432d-a354-710237f4e9b6 None admin] PATH_INFO: `/v3/projects/cd69c40698884b3192bb19ba6968842d` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:49.604099 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71d49e6d-4ee4-432d-a354-710237f4e9b6 None admin] RBAC: Authorizing `identity:get_project(project_id=cd69c40698884b3192bb19ba6968842d)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:49.605571 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71d49e6d-4ee4-432d-a354-710237f4e9b6 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:49.607612 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 192/396] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:41:49 2022] GET /identity/v3/projects/cd69c40698884b3192bb19ba6968842d => generated 298 bytes in 298 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:49.614758 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d49e6d-4ee4-432d-a354-710237f4e9b6 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:49.634157 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a295d80a-2f18-479c-b501-71f41df7d874 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:49.635520 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a295d80a-2f18-479c-b501-71f41df7d874 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:49.635946 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a295d80a-2f18-479c-b501-71f41df7d874 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:49.636757 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a295d80a-2f18-479c-b501-71f41df7d874 None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:49.637880 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a295d80a-2f18-479c-b501-71f41df7d874 None admin] RBAC: Adding query filter params (name=member) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:49.638346 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a295d80a-2f18-479c-b501-71f41df7d874 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:49.640137 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a295d80a-2f18-479c-b501-71f41df7d874 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:49.651539 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 205/397] 10.222.0.52 () {56 vars in 1101 bytes} [Sun Sep 25 00:41:49 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 297 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:49.653380 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd73d30-4125-493e-9d8b-65b479873f00 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:49.654195 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd73d30-4125-493e-9d8b-65b479873f00 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:49.654885 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd73d30-4125-493e-9d8b-65b479873f00 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:49.656540 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 206/398] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:49.657557 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd73d30-4125-493e-9d8b-65b479873f00 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:49.739769 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d49e6d-4ee4-432d-a354-710237f4e9b6 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:49.784083 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04015dc6-5d3d-452b-b0b1-f5aecc1158ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:49.785243 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04015dc6-5d3d-452b-b0b1-f5aecc1158ce None admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:49.785888 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04015dc6-5d3d-452b-b0b1-f5aecc1158ce None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:49.786399 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04015dc6-5d3d-452b-b0b1-f5aecc1158ce None admin] PATH_INFO: `/v3/projects/cd69c40698884b3192bb19ba6968842d/groups/f52544847fbb4d61b602bdeadf58d9d6/roles/a917909d7aa84d00a841c77e7e7bdc8a` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:49.789147 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04015dc6-5d3d-452b-b0b1-f5aecc1158ce None admin] RBAC: Authorizing `identity:create_grant(project_id=cd69c40698884b3192bb19ba6968842d, group_id=f52544847fbb4d61b602bdeadf58d9d6, role_id=a917909d7aa84d00a841c77e7e7bdc8a)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:49.789302 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd73d30-4125-493e-9d8b-65b479873f00 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:49.791382 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04015dc6-5d3d-452b-b0b1-f5aecc1158ce None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:49.808519 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 193/399] 10.222.0.52 () {58 vars in 1442 bytes} [Sun Sep 25 00:41:49 2022] PUT /identity/v3/projects/cd69c40698884b3192bb19ba6968842d/groups/f52544847fbb4d61b602bdeadf58d9d6/roles/a917909d7aa84d00a841c77e7e7bdc8a => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:41:49.810526 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e00beff2-c2bf-4566-927b-de2af7930ce3 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:49.811068 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e00beff2-c2bf-4566-927b-de2af7930ce3 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:49.811740 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e00beff2-c2bf-4566-927b-de2af7930ce3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:49.832052 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11b61f08-b0e4-4d15-b2f5-5cb0e5576182 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:49.832671 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b61f08-b0e4-4d15-b2f5-5cb0e5576182 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:49.832946 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b61f08-b0e4-4d15-b2f5-5cb0e5576182 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:49.833211 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b61f08-b0e4-4d15-b2f5-5cb0e5576182 None admin] PATH_INFO: `/v3/groups/f52544847fbb4d61b602bdeadf58d9d6` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:49.834605 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11b61f08-b0e4-4d15-b2f5-5cb0e5576182 None admin] RBAC: Authorizing `identity:get_group(group_id=f52544847fbb4d61b602bdeadf58d9d6)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:49.835653 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-e00beff2-c2bf-4566-927b-de2af7930ce3 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:49.836161 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11b61f08-b0e4-4d15-b2f5-5cb0e5576182 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:49.838016 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-e00beff2-c2bf-4566-927b-de2af7930ce3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:49.838919 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 207/400] 10.222.0.52 () {56 vars in 1180 bytes} [Sun Sep 25 00:41:49 2022] GET /identity/v3/groups/f52544847fbb4d61b602bdeadf58d9d6 => generated 226 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:49.845161 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11b61f08-b0e4-4d15-b2f5-5cb0e5576182 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:49.862006 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:49.864148 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:49.957042 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 194/401] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:41:49.958738 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca3b4c2-b629-48b5-876c-eea273e214f5 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:49.959248 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca3b4c2-b629-48b5-876c-eea273e214f5 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:49.959716 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca3b4c2-b629-48b5-876c-eea273e214f5 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:49.961888 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 195/402] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:49.969314 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af131bb9-80e3-4ff8-a690-df47841fc6a5 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:49.969314 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af131bb9-80e3-4ff8-a690-df47841fc6a5 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:49.969314 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af131bb9-80e3-4ff8-a690-df47841fc6a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:49.988519 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-af131bb9-80e3-4ff8-a690-df47841fc6a5 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:49.989840 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-af131bb9-80e3-4ff8-a690-df47841fc6a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:50.019755 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11b61f08-b0e4-4d15-b2f5-5cb0e5576182 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:50.069220 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be50928-c597-4fd3-8af7-4d06eeceff16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:50.070117 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be50928-c597-4fd3-8af7-4d06eeceff16 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:50.070607 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be50928-c597-4fd3-8af7-4d06eeceff16 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:50.071686 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be50928-c597-4fd3-8af7-4d06eeceff16 None admin] PATH_INFO: `/v3/projects/86aea44da5fb4bdeb527c7ff257c5d43` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:50.073065 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5be50928-c597-4fd3-8af7-4d06eeceff16 None admin] RBAC: Authorizing `identity:get_project(project_id=86aea44da5fb4bdeb527c7ff257c5d43)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:50.074683 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5be50928-c597-4fd3-8af7-4d06eeceff16 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:50.076670 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 208/403] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:41:49 2022] GET /identity/v3/projects/86aea44da5fb4bdeb527c7ff257c5d43 => generated 302 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:50.082512 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be50928-c597-4fd3-8af7-4d06eeceff16 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:50.083726 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 196/404] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 25 00:41:50.177166 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:50.201341 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be50928-c597-4fd3-8af7-4d06eeceff16 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:50.245048 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de334a01-c993-4730-851e-61fac266f527 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:50.245962 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de334a01-c993-4730-851e-61fac266f527 None admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:50.246377 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de334a01-c993-4730-851e-61fac266f527 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:50.246782 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de334a01-c993-4730-851e-61fac266f527 None admin] PATH_INFO: `/v3/projects/86aea44da5fb4bdeb527c7ff257c5d43/groups/f52544847fbb4d61b602bdeadf58d9d6/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:50.248972 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de334a01-c993-4730-851e-61fac266f527 None admin] RBAC: Authorizing `identity:create_grant(project_id=86aea44da5fb4bdeb527c7ff257c5d43, group_id=f52544847fbb4d61b602bdeadf58d9d6, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:50.250576 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de334a01-c993-4730-851e-61fac266f527 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:50.277578 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 209/405] 10.222.0.52 () {58 vars in 1442 bytes} [Sun Sep 25 00:41:50 2022] PUT /identity/v3/projects/86aea44da5fb4bdeb527c7ff257c5d43/groups/f52544847fbb4d61b602bdeadf58d9d6/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 00:41:50.307287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af131bb9-80e3-4ff8-a690-df47841fc6a5 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:50.394418 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de334a01-c993-4730-851e-61fac266f527 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:50.400993 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:50.443614 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af131bb9-80e3-4ff8-a690-df47841fc6a5 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:50.523540 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de334a01-c993-4730-851e-61fac266f527 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:50.525471 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7e0e315-8724-481e-8a8c-2ebac73c0b57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:50.526135 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e0e315-8724-481e-8a8c-2ebac73c0b57 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:50.526409 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e0e315-8724-481e-8a8c-2ebac73c0b57 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:50.526676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e0e315-8724-481e-8a8c-2ebac73c0b57 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:50.551640 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7e0e315-8724-481e-8a8c-2ebac73c0b57 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:41:50.557166 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-b7e0e315-8724-481e-8a8c-2ebac73c0b57 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 00:41:50.557166 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:41:50.557166 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:41:50.557166 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:41:50.557166 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:41:50.557166 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:41:50.557166 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:41:50.557166 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:41:50.557166 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:41:50.557166 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:41:50.557166 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:41:50.557166 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:41:50.557166 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:41:50.557166 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:41:50.557166 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:41:50.557166 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:41:50.558099 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:41:50.558099 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:41:50.558099 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:41:50.558099 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:41:50.558099 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:41:50.558099 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:41:50.558099 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:41:50.558099 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:41:50.558099 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:41:50.558099 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:41:50.558099 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:41:50.558099 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:41:50.558099 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:41:50.558099 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:41:50.558099 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:41:50.558099 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:41:50.558931 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:41:50.558931 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:41:50.558931 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:41:50.558931 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:41:50.558931 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:41:50.558931 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:41:50.558931 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:41:50.558931 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:41:50.558931 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:41:50.558931 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:41:50.558931 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 00:41:50.558931 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:41:50.564932 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 197/406] 10.222.0.52 () {56 vars in 1114 bytes} [Sun Sep 25 00:41:50 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 258 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:41:50.574390 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7e0e315-8724-481e-8a8c-2ebac73c0b57 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:50.574909 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e428bd3-8aeb-4aec-9567-d8e5c5a049e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:50.575580 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e428bd3-8aeb-4aec-9567-d8e5c5a049e1 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:50.575854 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e428bd3-8aeb-4aec-9567-d8e5c5a049e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:50.576116 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e428bd3-8aeb-4aec-9567-d8e5c5a049e1 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:50.598506 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e428bd3-8aeb-4aec-9567-d8e5c5a049e1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:41:50.604518 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-3e428bd3-8aeb-4aec-9567-d8e5c5a049e1 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:41:50.604518 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:41:50.604518 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:41:50.604518 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:41:50.604518 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:41:50.604518 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:41:50.604518 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:41:50.604518 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:41:50.604518 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:41:50.604518 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:41:50.604518 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:41:50.604518 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:41:50.604518 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:41:50.604518 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:41:50.604518 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:41:50.604518 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:41:50.605536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:41:50.605536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:41:50.605536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:41:50.605536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:41:50.605536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:41:50.605536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:41:50.605536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:41:50.605536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:41:50.605536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:41:50.605536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:41:50.605536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:41:50.605536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:41:50.605536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:41:50.605536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:41:50.605536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:41:50.605536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:41:50.607543 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:41:50.607543 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:41:50.607543 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:41:50.607543 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:41:50.607543 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:41:50.607543 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:41:50.607543 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:41:50.607543 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:41:50.607543 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:41:50.607543 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:41:50.607543 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:41:50.607543 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:41:50.607543 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 210/407] 10.222.0.52 () {56 vars in 1096 bytes} [Sun Sep 25 00:41:50 2022] GET /identity/v3/roles/admin => generated 83 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:41:50.610032 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:50.613568 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e428bd3-8aeb-4aec-9567-d8e5c5a049e1 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:50.722190 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e428bd3-8aeb-4aec-9567-d8e5c5a049e1 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:50.758807 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce365e3d-0550-4e12-b37d-41c37b55a6f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:50.759681 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce365e3d-0550-4e12-b37d-41c37b55a6f9 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:50.759966 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce365e3d-0550-4e12-b37d-41c37b55a6f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:50.760235 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce365e3d-0550-4e12-b37d-41c37b55a6f9 None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:50.760911 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce365e3d-0550-4e12-b37d-41c37b55a6f9 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:50.761214 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce365e3d-0550-4e12-b37d-41c37b55a6f9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:50.762651 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce365e3d-0550-4e12-b37d-41c37b55a6f9 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:50.771844 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 211/408] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:41:50 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:50.777707 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce365e3d-0550-4e12-b37d-41c37b55a6f9 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:50.789106 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7e0e315-8724-481e-8a8c-2ebac73c0b57 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:50.840892 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:50.885183 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce365e3d-0550-4e12-b37d-41c37b55a6f9 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:50.890734 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ffff76-23d8-4ad2-9e8e-2d0cc886090e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:50.891366 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ffff76-23d8-4ad2-9e8e-2d0cc886090e None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:50.891644 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ffff76-23d8-4ad2-9e8e-2d0cc886090e None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:50.891915 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ffff76-23d8-4ad2-9e8e-2d0cc886090e None admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:50.892596 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79ffff76-23d8-4ad2-9e8e-2d0cc886090e None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:50.892900 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79ffff76-23d8-4ad2-9e8e-2d0cc886090e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:50.894387 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79ffff76-23d8-4ad2-9e8e-2d0cc886090e None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:50.906591 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:50.919649 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 198/409] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:41:50 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 345 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:50.930092 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ffff76-23d8-4ad2-9e8e-2d0cc886090e None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:50.930494 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb4a399-bd2a-4898-b7c4-5c17f8cadcbc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:50.931155 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb4a399-bd2a-4898-b7c4-5c17f8cadcbc None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:50.931434 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb4a399-bd2a-4898-b7c4-5c17f8cadcbc None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:50.931706 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb4a399-bd2a-4898-b7c4-5c17f8cadcbc None admin] PATH_INFO: `/v3/groups/3593b329027d4ed7abe6cc428574877a` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:50.932893 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fb4a399-bd2a-4898-b7c4-5c17f8cadcbc None admin] RBAC: Authorizing `identity:get_group(group_id=3593b329027d4ed7abe6cc428574877a)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:50.934408 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fb4a399-bd2a-4898-b7c4-5c17f8cadcbc None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:50.936337 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 212/410] 10.222.0.52 () {56 vars in 1180 bytes} [Sun Sep 25 00:41:50 2022] GET /identity/v3/groups/3593b329027d4ed7abe6cc428574877a => generated 229 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:41:50.941803 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb4a399-bd2a-4898-b7c4-5c17f8cadcbc None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:51.054148 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb4a399-bd2a-4898-b7c4-5c17f8cadcbc None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:51.088249 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:51.099802 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d016f75c-04bb-459c-b21f-d1cd8c2f7c93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:51.100470 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d016f75c-04bb-459c-b21f-d1cd8c2f7c93 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:51.100745 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d016f75c-04bb-459c-b21f-d1cd8c2f7c93 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:51.101013 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d016f75c-04bb-459c-b21f-d1cd8c2f7c93 None admin] PATH_INFO: `/v3/projects/c8e32a9668254c2e96354fd797883364` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:51.102427 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d016f75c-04bb-459c-b21f-d1cd8c2f7c93 None admin] RBAC: Authorizing `identity:get_project(project_id=c8e32a9668254c2e96354fd797883364)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:51.104682 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d016f75c-04bb-459c-b21f-d1cd8c2f7c93 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:51.106665 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 213/411] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:41:50 2022] GET /identity/v3/projects/c8e32a9668254c2e96354fd797883364 => generated 344 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:41:51.107814 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a1c095-1c37-4aa0-acbd-17b75a790c4a None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:51.108092 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a1c095-1c37-4aa0-acbd-17b75a790c4a None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:51.108364 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a1c095-1c37-4aa0-acbd-17b75a790c4a None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:51.109893 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 214/412] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:51.114700 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a1c095-1c37-4aa0-acbd-17b75a790c4a None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:51.118654 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ffff76-23d8-4ad2-9e8e-2d0cc886090e None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:51.155799 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d18a995-f4b6-4d9a-b1c6-e0eb41c97ee5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:51.156426 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d18a995-f4b6-4d9a-b1c6-e0eb41c97ee5 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:51.156698 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d18a995-f4b6-4d9a-b1c6-e0eb41c97ee5 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:51.156962 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d18a995-f4b6-4d9a-b1c6-e0eb41c97ee5 None admin] PATH_INFO: `/v3/groups/f52544847fbb4d61b602bdeadf58d9d6` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:51.158153 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d18a995-f4b6-4d9a-b1c6-e0eb41c97ee5 None admin] RBAC: Authorizing `identity:get_group(group_id=f52544847fbb4d61b602bdeadf58d9d6)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:51.160141 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d18a995-f4b6-4d9a-b1c6-e0eb41c97ee5 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:51.161827 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 199/413] 10.222.0.52 () {56 vars in 1180 bytes} [Sun Sep 25 00:41:50 2022] GET /identity/v3/groups/f52544847fbb4d61b602bdeadf58d9d6 => generated 226 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:51.163170 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b490a68-d0c2-4d8d-a2f0-2b801e04f7f4 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:51.163447 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b490a68-d0c2-4d8d-a2f0-2b801e04f7f4 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:51.163716 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b490a68-d0c2-4d8d-a2f0-2b801e04f7f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:51.184910 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-2b490a68-d0c2-4d8d-a2f0-2b801e04f7f4 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:51.186190 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-2b490a68-d0c2-4d8d-a2f0-2b801e04f7f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:51.239338 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a1c095-1c37-4aa0-acbd-17b75a790c4a None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:51.256021 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:51.267847 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 200/414] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:41:51.269351 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b490a68-d0c2-4d8d-a2f0-2b801e04f7f4 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:51.297578 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed860aec-2209-45e4-9bee-c021744bde67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:51.298299 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed860aec-2209-45e4-9bee-c021744bde67 None admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:51.298567 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed860aec-2209-45e4-9bee-c021744bde67 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:51.300302 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed860aec-2209-45e4-9bee-c021744bde67 None admin] PATH_INFO: `/v3/projects/c8e32a9668254c2e96354fd797883364/groups/3593b329027d4ed7abe6cc428574877a/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:51.303151 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed860aec-2209-45e4-9bee-c021744bde67 None admin] RBAC: Authorizing `identity:create_grant(project_id=c8e32a9668254c2e96354fd797883364, group_id=3593b329027d4ed7abe6cc428574877a, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:51.306211 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed860aec-2209-45e4-9bee-c021744bde67 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:51.330967 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 215/415] 10.222.0.52 () {58 vars in 1442 bytes} [Sun Sep 25 00:41:51 2022] PUT /identity/v3/projects/c8e32a9668254c2e96354fd797883364/groups/3593b329027d4ed7abe6cc428574877a/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:41:51.409744 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b490a68-d0c2-4d8d-a2f0-2b801e04f7f4 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:51.457716 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75288ee-c76d-4c30-92c6-39225e40ca18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:51.458422 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75288ee-c76d-4c30-92c6-39225e40ca18 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:51.458698 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75288ee-c76d-4c30-92c6-39225e40ca18 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:51.458959 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75288ee-c76d-4c30-92c6-39225e40ca18 None admin] PATH_INFO: `/v3/projects/86aea44da5fb4bdeb527c7ff257c5d43` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:51.460171 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d75288ee-c76d-4c30-92c6-39225e40ca18 None admin] RBAC: Authorizing `identity:get_project(project_id=86aea44da5fb4bdeb527c7ff257c5d43)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:51.461988 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d75288ee-c76d-4c30-92c6-39225e40ca18 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:51.465796 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 201/416] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:41:51 2022] GET /identity/v3/projects/86aea44da5fb4bdeb527c7ff257c5d43 => generated 302 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:51.469768 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed860aec-2209-45e4-9bee-c021744bde67 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:51.574086 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:51.633977 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed860aec-2209-45e4-9bee-c021744bde67 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:51.677040 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb0355e-491e-4109-a4e3-35b1dd4aee57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:51.677778 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb0355e-491e-4109-a4e3-35b1dd4aee57 None admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:51.678052 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb0355e-491e-4109-a4e3-35b1dd4aee57 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:51.678315 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb0355e-491e-4109-a4e3-35b1dd4aee57 None admin] PATH_INFO: `/v3/projects/86aea44da5fb4bdeb527c7ff257c5d43/groups/f52544847fbb4d61b602bdeadf58d9d6/roles/a917909d7aa84d00a841c77e7e7bdc8a` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:51.680637 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb0355e-491e-4109-a4e3-35b1dd4aee57 None admin] RBAC: Authorizing `identity:create_grant(project_id=86aea44da5fb4bdeb527c7ff257c5d43, group_id=f52544847fbb4d61b602bdeadf58d9d6, role_id=a917909d7aa84d00a841c77e7e7bdc8a)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:51.682164 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb0355e-491e-4109-a4e3-35b1dd4aee57 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:51.709057 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 216/417] 10.222.0.52 () {58 vars in 1442 bytes} [Sun Sep 25 00:41:51 2022] PUT /identity/v3/projects/86aea44da5fb4bdeb527c7ff257c5d43/groups/f52544847fbb4d61b602bdeadf58d9d6/roles/a917909d7aa84d00a841c77e7e7bdc8a => generated 0 bytes in 239 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:41:51.709327 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75288ee-c76d-4c30-92c6-39225e40ca18 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:51.795778 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:51.851877 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75288ee-c76d-4c30-92c6-39225e40ca18 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:51.906520 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d2857de-41e9-48f1-b41b-ea5ff7a4c64a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:51.907167 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2857de-41e9-48f1-b41b-ea5ff7a4c64a None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:51.907443 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2857de-41e9-48f1-b41b-ea5ff7a4c64a None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:51.907785 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2857de-41e9-48f1-b41b-ea5ff7a4c64a None admin] PATH_INFO: `/v3/roles/member` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:51.929717 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d2857de-41e9-48f1-b41b-ea5ff7a4c64a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:41:51.935392 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-3d2857de-41e9-48f1-b41b-ea5ff7a4c64a None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 25 00:41:51.935392 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:41:51.935392 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:41:51.935392 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:41:51.935392 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:41:51.935392 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:41:51.935392 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:41:51.935392 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:41:51.935392 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:41:51.935392 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:41:51.935392 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:41:51.935392 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:41:51.935392 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:41:51.935392 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:41:51.935392 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:41:51.935392 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:41:51.936364 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:41:51.936364 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:41:51.936364 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:41:51.936364 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:41:51.936364 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:41:51.936364 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:41:51.936364 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:41:51.936364 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:41:51.936364 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:41:51.936364 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:41:51.936364 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:41:51.936364 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:41:51.936364 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:41:51.936364 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:41:51.936364 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:41:51.936364 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:41:51.937873 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:41:51.937873 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:41:51.937873 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:41:51.937873 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:41:51.937873 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:41:51.937873 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:41:51.937873 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:41:51.937873 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:41:51.937873 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:41:51.937873 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:41:51.937873 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 25 00:41:51.937873 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:41:51.937873 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 202/418] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:41:51 2022] GET /identity/v3/roles/member => generated 84 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:41:51.942489 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb0355e-491e-4109-a4e3-35b1dd4aee57 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:52.095234 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb0355e-491e-4109-a4e3-35b1dd4aee57 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:52.119911 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:52.148771 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c013c171-4421-4563-961c-6ada08fe1431 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:52.150219 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c013c171-4421-4563-961c-6ada08fe1431 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:52.150491 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c013c171-4421-4563-961c-6ada08fe1431 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:52.150759 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c013c171-4421-4563-961c-6ada08fe1431 None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:52.151431 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c013c171-4421-4563-961c-6ada08fe1431 None admin] RBAC: Adding query filter params (name=member) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:52.151727 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c013c171-4421-4563-961c-6ada08fe1431 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:52.157194 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c013c171-4421-4563-961c-6ada08fe1431 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:52.167687 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 217/419] 10.222.0.52 () {56 vars in 1101 bytes} [Sun Sep 25 00:41:51 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:52.173092 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d2857de-41e9-48f1-b41b-ea5ff7a4c64a None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:52.307766 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d2857de-41e9-48f1-b41b-ea5ff7a4c64a None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:52.341044 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:52.355724 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16da8dfb-6596-4120-8a1b-8f37857ba0af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:52.356681 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16da8dfb-6596-4120-8a1b-8f37857ba0af None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:52.357182 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16da8dfb-6596-4120-8a1b-8f37857ba0af None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:52.357669 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16da8dfb-6596-4120-8a1b-8f37857ba0af None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:52.453340 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16da8dfb-6596-4120-8a1b-8f37857ba0af None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:52.455182 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16da8dfb-6596-4120-8a1b-8f37857ba0af None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:52.595675 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 203/420] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:41:52 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 423 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 00:41:52.601375 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c013c171-4421-4563-961c-6ada08fe1431 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:52.741668 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c90eb0-511b-43c9-8f25-ea90e6aae949 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:52.742016 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c90eb0-511b-43c9-8f25-ea90e6aae949 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:52.742331 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c90eb0-511b-43c9-8f25-ea90e6aae949 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:52.743274 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c013c171-4421-4563-961c-6ada08fe1431 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:52.744614 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 204/421] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:52.750335 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc0a3ed-4833-4262-870b-dc41ee284421 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:52.750613 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc0a3ed-4833-4262-870b-dc41ee284421 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:52.750885 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc0a3ed-4833-4262-870b-dc41ee284421 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:52.788095 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-ffc0a3ed-4833-4262-870b-dc41ee284421 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:52.789326 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-ffc0a3ed-4833-4262-870b-dc41ee284421 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:52.792826 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfbdfca3-29a2-434b-8c24-498b16bafa6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:52.793495 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfbdfca3-29a2-434b-8c24-498b16bafa6a None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:52.793766 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfbdfca3-29a2-434b-8c24-498b16bafa6a None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:52.794033 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfbdfca3-29a2-434b-8c24-498b16bafa6a None admin] PATH_INFO: `/v3/projects/cd69c40698884b3192bb19ba6968842d` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:52.796818 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfbdfca3-29a2-434b-8c24-498b16bafa6a None admin] RBAC: Authorizing `identity:get_project(project_id=cd69c40698884b3192bb19ba6968842d)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:52.798289 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfbdfca3-29a2-434b-8c24-498b16bafa6a None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:52.805808 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 218/422] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:41:52 2022] GET /identity/v3/projects/cd69c40698884b3192bb19ba6968842d => generated 298 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:52.806784 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfbdfca3-29a2-434b-8c24-498b16bafa6a None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:52.850142 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:52.913644 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:52.959020 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 205/423] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:41:52.974153 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfbdfca3-29a2-434b-8c24-498b16bafa6a None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:52.997638 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73893e6-2e38-4cf9-b502-be5f92794db9 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:52.998246 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73893e6-2e38-4cf9-b502-be5f92794db9 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:52.998737 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73893e6-2e38-4cf9-b502-be5f92794db9 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:53.000867 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 206/424] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:52 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 00:41:53.006858 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31055e02-2246-4a53-b2e1-2c3b7f92adb8 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:53.007377 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31055e02-2246-4a53-b2e1-2c3b7f92adb8 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:53.007869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31055e02-2246-4a53-b2e1-2c3b7f92adb8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:53.019129 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd77c8d-b762-453b-ba2e-ca71f7ea89b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:53.019760 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd77c8d-b762-453b-ba2e-ca71f7ea89b2 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:53.020029 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd77c8d-b762-453b-ba2e-ca71f7ea89b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:53.020292 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd77c8d-b762-453b-ba2e-ca71f7ea89b2 None admin] PATH_INFO: `/v3/groups/f52544847fbb4d61b602bdeadf58d9d6` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:53.022640 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bd77c8d-b762-453b-ba2e-ca71f7ea89b2 None admin] RBAC: Authorizing `identity:get_group(group_id=f52544847fbb4d61b602bdeadf58d9d6)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:53.024229 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bd77c8d-b762-453b-ba2e-ca71f7ea89b2 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:53.030145 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 219/425] 10.222.0.52 () {56 vars in 1180 bytes} [Sun Sep 25 00:41:52 2022] GET /identity/v3/groups/f52544847fbb4d61b602bdeadf58d9d6 => generated 226 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:53.035961 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd77c8d-b762-453b-ba2e-ca71f7ea89b2 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:53.050314 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-31055e02-2246-4a53-b2e1-2c3b7f92adb8 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:53.051577 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-31055e02-2246-4a53-b2e1-2c3b7f92adb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:53.146457 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd77c8d-b762-453b-ba2e-ca71f7ea89b2 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:53.184359 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf30697f-b555-4a4c-b042-adcb800af1fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:53.185117 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf30697f-b555-4a4c-b042-adcb800af1fd None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:53.185425 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf30697f-b555-4a4c-b042-adcb800af1fd None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:53.185693 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf30697f-b555-4a4c-b042-adcb800af1fd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:53.186409 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf30697f-b555-4a4c-b042-adcb800af1fd None admin] RBAC: Adding query filter params (group.id=f52544847fbb4d61b602bdeadf58d9d6, role.id=40f54f5ac0994c51960ac413da8d2cc9, scope.project.id=cd69c40698884b3192bb19ba6968842d) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:53.186711 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf30697f-b555-4a4c-b042-adcb800af1fd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:53.187176 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:53.193963 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf30697f-b555-4a4c-b042-adcb800af1fd None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:53.204700 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 220/426] 10.222.0.52 () {56 vars in 1376 bytes} [Sun Sep 25 00:41:53 2022] GET /identity/v3/role_assignments?group.id=f52544847fbb4d61b602bdeadf58d9d6&scope.project.id=cd69c40698884b3192bb19ba6968842d&role.id=40f54f5ac0994c51960ac413da8d2cc9 => generated 611 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:41:53.210882 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 207/427] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 25 00:41:53.274026 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf30697f-b555-4a4c-b042-adcb800af1fd None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:53.422572 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf30697f-b555-4a4c-b042-adcb800af1fd None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:53.465153 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7fd17eb-c8d7-4842-8b0d-aea8a9728d94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:53.466195 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7fd17eb-c8d7-4842-8b0d-aea8a9728d94 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:53.466663 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7fd17eb-c8d7-4842-8b0d-aea8a9728d94 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:53.467148 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7fd17eb-c8d7-4842-8b0d-aea8a9728d94 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:53.486341 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7fd17eb-c8d7-4842-8b0d-aea8a9728d94 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:41:53.492007 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-c7fd17eb-c8d7-4842-8b0d-aea8a9728d94 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 00:41:53.492007 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:41:53.492007 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:41:53.492007 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:41:53.492007 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:41:53.492007 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:41:53.492007 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:41:53.492007 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:41:53.492007 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:41:53.492007 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:41:53.492007 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:41:53.492007 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:41:53.492007 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:41:53.492007 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:41:53.492007 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:41:53.492007 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:41:53.493549 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:41:53.493549 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:41:53.493549 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:41:53.493549 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:41:53.493549 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:41:53.493549 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:41:53.493549 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:41:53.493549 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:41:53.493549 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:41:53.493549 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:41:53.493549 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:41:53.493549 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:41:53.493549 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:41:53.493549 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:41:53.493549 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:41:53.493549 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:41:53.494778 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:41:53.494778 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:41:53.494778 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:41:53.494778 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:41:53.494778 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:41:53.494778 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:41:53.494778 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:41:53.494778 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:41:53.494778 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:41:53.494778 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:41:53.494778 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 00:41:53.494778 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:41:53.494778 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 221/428] 10.222.0.52 () {56 vars in 1114 bytes} [Sun Sep 25 00:41:53 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:41:53.500296 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31055e02-2246-4a53-b2e1-2c3b7f92adb8 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:53.542441 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7fd17eb-c8d7-4842-8b0d-aea8a9728d94 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:53.595452 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:53.622641 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:53.654217 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31055e02-2246-4a53-b2e1-2c3b7f92adb8 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:53.688735 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7fd17eb-c8d7-4842-8b0d-aea8a9728d94 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:53.703227 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-842a04fe-d0da-453e-af7b-382eff1d8252 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:53.703840 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-842a04fe-d0da-453e-af7b-382eff1d8252 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:53.704345 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-842a04fe-d0da-453e-af7b-382eff1d8252 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:53.704625 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-842a04fe-d0da-453e-af7b-382eff1d8252 None admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:53.705800 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-842a04fe-d0da-453e-af7b-382eff1d8252 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:53.706104 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-842a04fe-d0da-453e-af7b-382eff1d8252 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:53.708004 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-842a04fe-d0da-453e-af7b-382eff1d8252 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:53.720578 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 208/429] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:41:53 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:53.729090 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-842a04fe-d0da-453e-af7b-382eff1d8252 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:53.729655 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfbf316e-7e1e-4102-85b9-0ff5b742e97b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:53.730305 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfbf316e-7e1e-4102-85b9-0ff5b742e97b None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:53.730583 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfbf316e-7e1e-4102-85b9-0ff5b742e97b None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:53.730849 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfbf316e-7e1e-4102-85b9-0ff5b742e97b None admin] PATH_INFO: `/v3/roles/member` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:53.743932 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfbf316e-7e1e-4102-85b9-0ff5b742e97b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:41:53.749662 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-dfbf316e-7e1e-4102-85b9-0ff5b742e97b None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 25 00:41:53.749662 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:41:53.749662 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:41:53.749662 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:41:53.749662 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:41:53.749662 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:41:53.749662 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:41:53.749662 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:41:53.749662 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:41:53.749662 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:41:53.749662 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:41:53.749662 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:41:53.749662 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:41:53.749662 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:41:53.749662 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:41:53.749662 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:41:53.750592 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:41:53.750592 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:41:53.750592 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:41:53.750592 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:41:53.750592 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:41:53.750592 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:41:53.750592 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:41:53.750592 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:41:53.750592 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:41:53.750592 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:41:53.750592 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:41:53.750592 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:41:53.750592 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:41:53.750592 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:41:53.750592 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:41:53.750592 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:41:53.752003 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:41:53.752003 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:41:53.752003 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:41:53.752003 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:41:53.752003 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:41:53.752003 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:41:53.752003 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:41:53.752003 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:41:53.752003 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:41:53.752003 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:41:53.752003 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 25 00:41:53.752003 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:41:53.752003 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 222/430] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:41:53 2022] GET /identity/v3/roles/member => generated 84 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:41:53.756575 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfbf316e-7e1e-4102-85b9-0ff5b742e97b None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:53.858574 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-842a04fe-d0da-453e-af7b-382eff1d8252 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:53.869840 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:53.874575 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfbf316e-7e1e-4102-85b9-0ff5b742e97b None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:53.896266 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-042682a4-da53-466b-b41f-9b1cff40bd66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:53.897103 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042682a4-da53-466b-b41f-9b1cff40bd66 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:53.897344 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042682a4-da53-466b-b41f-9b1cff40bd66 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:53.897539 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042682a4-da53-466b-b41f-9b1cff40bd66 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:53.912716 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673fbfd4-368b-4fcf-b84e-ac3acb2c96f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:53.913356 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673fbfd4-368b-4fcf-b84e-ac3acb2c96f0 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:53.913635 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673fbfd4-368b-4fcf-b84e-ac3acb2c96f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:53.913908 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673fbfd4-368b-4fcf-b84e-ac3acb2c96f0 None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:53.915119 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-673fbfd4-368b-4fcf-b84e-ac3acb2c96f0 None admin] RBAC: Adding query filter params (name=member) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:53.915447 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-673fbfd4-368b-4fcf-b84e-ac3acb2c96f0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:53.917537 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-673fbfd4-368b-4fcf-b84e-ac3acb2c96f0 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:53.919676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:53.927035 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 223/431] 10.222.0.52 () {56 vars in 1101 bytes} [Sun Sep 25 00:41:53 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:41:53.935356 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673fbfd4-368b-4fcf-b84e-ac3acb2c96f0 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:53.997162 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-042682a4-da53-466b-b41f-9b1cff40bd66 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:53.998653 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-042682a4-da53-466b-b41f-9b1cff40bd66 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:54.049832 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673fbfd4-368b-4fcf-b84e-ac3acb2c96f0 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:54.089902 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0cd5f12-5df5-4243-a9ee-8452067874d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:54.090515 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0cd5f12-5df5-4243-a9ee-8452067874d3 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:54.090791 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0cd5f12-5df5-4243-a9ee-8452067874d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:54.091061 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0cd5f12-5df5-4243-a9ee-8452067874d3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:54.110869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 209/432] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:41:53 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 383 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 00:41:54.117017 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-042682a4-da53-466b-b41f-9b1cff40bd66 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:54.125716 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:54.172561 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0cd5f12-5df5-4243-a9ee-8452067874d3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:54.174023 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:54.174831 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0cd5f12-5df5-4243-a9ee-8452067874d3 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:54.232684 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-042682a4-da53-466b-b41f-9b1cff40bd66 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:54.271552 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf1942c8-be14-4aa3-96b7-f27552d39f07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:54.272184 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1942c8-be14-4aa3-96b7-f27552d39f07 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:54.272452 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1942c8-be14-4aa3-96b7-f27552d39f07 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:54.272712 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1942c8-be14-4aa3-96b7-f27552d39f07 None admin] PATH_INFO: `/v3/projects/cd69c40698884b3192bb19ba6968842d` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:54.274420 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf1942c8-be14-4aa3-96b7-f27552d39f07 None admin] RBAC: Authorizing `identity:get_project(project_id=cd69c40698884b3192bb19ba6968842d)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:54.276241 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf1942c8-be14-4aa3-96b7-f27552d39f07 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:54.278080 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 210/433] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:41:54 2022] GET /identity/v3/projects/cd69c40698884b3192bb19ba6968842d => generated 298 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:41:54.279218 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8443532-b069-4181-8ff9-3bc6156c30cd None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:54.279487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8443532-b069-4181-8ff9-3bc6156c30cd None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:54.279749 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8443532-b069-4181-8ff9-3bc6156c30cd None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:54.281175 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 211/434] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:54 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:54.286917 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995fb010-8145-4c59-89d5-2927a8b4678f None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:54.287194 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995fb010-8145-4c59-89d5-2927a8b4678f None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:54.287462 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995fb010-8145-4c59-89d5-2927a8b4678f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:54.295543 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 224/435] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:41:53 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 360 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 00:41:54.296596 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0cd5f12-5df5-4243-a9ee-8452067874d3 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:54.311958 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-995fb010-8145-4c59-89d5-2927a8b4678f None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:54.313121 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-995fb010-8145-4c59-89d5-2927a8b4678f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:54.357273 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:54.386891 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 212/436] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:41:54.390258 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995fb010-8145-4c59-89d5-2927a8b4678f None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:54.404068 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0cd5f12-5df5-4243-a9ee-8452067874d3 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:54.455958 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438bf965-10f6-499d-8705-0e78cc5baf90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:54.457809 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438bf965-10f6-499d-8705-0e78cc5baf90 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:54.458327 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438bf965-10f6-499d-8705-0e78cc5baf90 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:54.459249 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438bf965-10f6-499d-8705-0e78cc5baf90 None admin] PATH_INFO: `/v3/groups/f52544847fbb4d61b602bdeadf58d9d6` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:54.461422 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438bf965-10f6-499d-8705-0e78cc5baf90 None admin] RBAC: Authorizing `identity:get_group(group_id=f52544847fbb4d61b602bdeadf58d9d6)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:54.462291 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:54.465519 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438bf965-10f6-499d-8705-0e78cc5baf90 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:54.468617 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 225/437] 10.222.0.52 () {56 vars in 1180 bytes} [Sun Sep 25 00:41:54 2022] GET /identity/v3/groups/f52544847fbb4d61b602bdeadf58d9d6 => generated 226 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:54.474388 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438bf965-10f6-499d-8705-0e78cc5baf90 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:54.538915 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995fb010-8145-4c59-89d5-2927a8b4678f None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:54.578133 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78580693-179b-43e4-afad-8913c6d5ab68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:54.579168 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78580693-179b-43e4-afad-8913c6d5ab68 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:54.579668 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78580693-179b-43e4-afad-8913c6d5ab68 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:54.580133 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78580693-179b-43e4-afad-8913c6d5ab68 None admin] PATH_INFO: `/v3/projects/86aea44da5fb4bdeb527c7ff257c5d43` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:54.581540 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78580693-179b-43e4-afad-8913c6d5ab68 None admin] RBAC: Authorizing `identity:get_project(project_id=86aea44da5fb4bdeb527c7ff257c5d43)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:54.583569 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78580693-179b-43e4-afad-8913c6d5ab68 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:54.585701 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 213/438] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:41:54 2022] GET /identity/v3/projects/86aea44da5fb4bdeb527c7ff257c5d43 => generated 302 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:54.587336 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a3ca61-a98c-4487-b12a-99d168774e13 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:54.587821 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a3ca61-a98c-4487-b12a-99d168774e13 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:54.588285 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a3ca61-a98c-4487-b12a-99d168774e13 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:54.589870 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 214/439] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:54.591842 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a3ca61-a98c-4487-b12a-99d168774e13 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:54.608602 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438bf965-10f6-499d-8705-0e78cc5baf90 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:54.639686 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:54.648370 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c725803-11da-4973-8f20-612b043eb46c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:54.649139 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c725803-11da-4973-8f20-612b043eb46c None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:54.649502 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c725803-11da-4973-8f20-612b043eb46c None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:54.649738 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c725803-11da-4973-8f20-612b043eb46c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:54.650446 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c725803-11da-4973-8f20-612b043eb46c None admin] RBAC: Adding query filter params (group.id=f52544847fbb4d61b602bdeadf58d9d6, role.id=a917909d7aa84d00a841c77e7e7bdc8a, scope.project.id=cd69c40698884b3192bb19ba6968842d) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:54.650747 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c725803-11da-4973-8f20-612b043eb46c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:54.652173 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c725803-11da-4973-8f20-612b043eb46c None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:54.663978 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 226/440] 10.222.0.52 () {56 vars in 1376 bytes} [Sun Sep 25 00:41:54 2022] GET /identity/v3/role_assignments?group.id=f52544847fbb4d61b602bdeadf58d9d6&scope.project.id=cd69c40698884b3192bb19ba6968842d&role.id=a917909d7aa84d00a841c77e7e7bdc8a => generated 611 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:41:54.663978 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b36de0-d973-4dc0-8f64-02038af04095 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:54.663978 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b36de0-d973-4dc0-8f64-02038af04095 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:54.663978 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b36de0-d973-4dc0-8f64-02038af04095 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:54.682991 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-d9b36de0-d973-4dc0-8f64-02038af04095 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:54.684241 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-d9b36de0-d973-4dc0-8f64-02038af04095 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:54.723770 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a3ca61-a98c-4487-b12a-99d168774e13 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:54.741923 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:54.765683 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 227/441] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:41:54.767233 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b36de0-d973-4dc0-8f64-02038af04095 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:54.769624 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077acfdb-5836-41ea-97f9-48c4c65a0902 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:54.770265 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077acfdb-5836-41ea-97f9-48c4c65a0902 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:54.770536 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077acfdb-5836-41ea-97f9-48c4c65a0902 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:54.770798 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077acfdb-5836-41ea-97f9-48c4c65a0902 None admin] PATH_INFO: `/v3/groups/f52544847fbb4d61b602bdeadf58d9d6` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:54.771987 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-077acfdb-5836-41ea-97f9-48c4c65a0902 None admin] RBAC: Authorizing `identity:get_group(group_id=f52544847fbb4d61b602bdeadf58d9d6)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:54.773548 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-077acfdb-5836-41ea-97f9-48c4c65a0902 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:54.781691 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 215/442] 10.222.0.52 () {56 vars in 1180 bytes} [Sun Sep 25 00:41:54 2022] GET /identity/v3/groups/f52544847fbb4d61b602bdeadf58d9d6 => generated 226 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:54.782180 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077acfdb-5836-41ea-97f9-48c4c65a0902 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:54.892803 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b36de0-d973-4dc0-8f64-02038af04095 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:54.913412 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077acfdb-5836-41ea-97f9-48c4c65a0902 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:54.924647 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:54.951080 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d78cb0-b5fc-4efc-810e-9f9b94fea63f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:54.951822 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d78cb0-b5fc-4efc-810e-9f9b94fea63f None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:54.952413 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d78cb0-b5fc-4efc-810e-9f9b94fea63f None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:54.952517 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d78cb0-b5fc-4efc-810e-9f9b94fea63f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:54.957955 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8b4f0ec-24db-49e6-9081-62b58433fac5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:54.958570 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b4f0ec-24db-49e6-9081-62b58433fac5 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:54.958842 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b4f0ec-24db-49e6-9081-62b58433fac5 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:54.959105 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b4f0ec-24db-49e6-9081-62b58433fac5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:54.959825 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8b4f0ec-24db-49e6-9081-62b58433fac5 None admin] RBAC: Adding query filter params (group.id=f52544847fbb4d61b602bdeadf58d9d6, role.id=40f54f5ac0994c51960ac413da8d2cc9, scope.project.id=86aea44da5fb4bdeb527c7ff257c5d43) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:54.960128 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8b4f0ec-24db-49e6-9081-62b58433fac5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:54.962238 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8b4f0ec-24db-49e6-9081-62b58433fac5 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:54.970478 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 216/443] 10.222.0.52 () {56 vars in 1376 bytes} [Sun Sep 25 00:41:54 2022] GET /identity/v3/role_assignments?group.id=f52544847fbb4d61b602bdeadf58d9d6&scope.project.id=86aea44da5fb4bdeb527c7ff257c5d43&role.id=40f54f5ac0994c51960ac413da8d2cc9 => generated 611 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:54.972378 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38d78cb0-b5fc-4efc-810e-9f9b94fea63f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:41:54.978262 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-38d78cb0-b5fc-4efc-810e-9f9b94fea63f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:41:54.978262 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:41:54.978262 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:41:54.978262 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:41:54.978262 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:41:54.978262 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:41:54.978262 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:41:54.978262 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:41:54.978262 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:41:54.978262 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:41:54.978262 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:41:54.978262 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:41:54.978262 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:41:54.978262 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:41:54.978262 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:41:54.978262 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:41:54.979251 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:41:54.979251 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:41:54.979251 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:41:54.979251 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:41:54.979251 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:41:54.979251 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:41:54.979251 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:41:54.979251 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:41:54.979251 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:41:54.979251 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:41:54.979251 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:41:54.979251 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:41:54.979251 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:41:54.979251 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:41:54.979251 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:41:54.979251 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:41:54.980131 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:41:54.980131 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:41:54.980131 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:41:54.980131 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:41:54.980131 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:41:54.980131 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:41:54.980131 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:41:54.980131 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:41:54.980131 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:41:54.980131 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:41:54.980131 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:41:54.980131 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:41:54.982221 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 228/444] 10.222.0.52 () {56 vars in 1096 bytes} [Sun Sep 25 00:41:54 2022] GET /identity/v3/roles/admin => generated 83 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:41:54.990804 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8b4f0ec-24db-49e6-9081-62b58433fac5 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:55.048184 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:55.052522 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d78cb0-b5fc-4efc-810e-9f9b94fea63f None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:55.127446 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8b4f0ec-24db-49e6-9081-62b58433fac5 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:55.173164 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34e97a27-49fe-4e98-a0fd-e9a766d5c4cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:55.174492 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e97a27-49fe-4e98-a0fd-e9a766d5c4cd None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:55.174947 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e97a27-49fe-4e98-a0fd-e9a766d5c4cd None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:55.175359 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e97a27-49fe-4e98-a0fd-e9a766d5c4cd None admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:55.176460 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34e97a27-49fe-4e98-a0fd-e9a766d5c4cd None admin] RBAC: Adding query filter params (name=admin) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:55.176919 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34e97a27-49fe-4e98-a0fd-e9a766d5c4cd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:55.179027 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34e97a27-49fe-4e98-a0fd-e9a766d5c4cd None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:55.187913 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d78cb0-b5fc-4efc-810e-9f9b94fea63f None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:55.188726 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 217/445] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:41:54 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:55.196412 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34e97a27-49fe-4e98-a0fd-e9a766d5c4cd None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:55.237323 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a4668e9-3779-43bb-890c-a2cf3f3bb4c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:55.238807 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4668e9-3779-43bb-890c-a2cf3f3bb4c6 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:55.238807 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4668e9-3779-43bb-890c-a2cf3f3bb4c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:55.238971 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4668e9-3779-43bb-890c-a2cf3f3bb4c6 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:55.250971 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a4668e9-3779-43bb-890c-a2cf3f3bb4c6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:41:55.255800 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-2a4668e9-3779-43bb-890c-a2cf3f3bb4c6 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 00:41:55.255800 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:41:55.255800 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:41:55.255800 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:41:55.255800 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:41:55.255800 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:41:55.255800 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:41:55.255800 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:41:55.255800 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:41:55.255800 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:41:55.255800 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:41:55.255800 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:41:55.255800 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:41:55.255800 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:41:55.255800 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:41:55.255800 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:41:55.256674 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:41:55.256674 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:41:55.256674 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:41:55.256674 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:41:55.256674 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:41:55.256674 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:41:55.256674 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:41:55.256674 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:41:55.256674 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:41:55.256674 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:41:55.256674 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:41:55.256674 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:41:55.256674 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:41:55.256674 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:41:55.256674 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:41:55.256674 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:41:55.257537 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:41:55.257537 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:41:55.257537 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:41:55.257537 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:41:55.257537 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:41:55.257537 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:41:55.257537 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:41:55.257537 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:41:55.257537 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:41:55.257537 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:41:55.257537 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 00:41:55.257537 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:41:55.257537 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 229/446] 10.222.0.52 () {56 vars in 1114 bytes} [Sun Sep 25 00:41:55 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:41:55.264093 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a4668e9-3779-43bb-890c-a2cf3f3bb4c6 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:55.315177 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:55.316438 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34e97a27-49fe-4e98-a0fd-e9a766d5c4cd None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:55.325448 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:55.358208 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa814824-6e18-40a3-b41b-9075474152e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:55.358208 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa814824-6e18-40a3-b41b-9075474152e5 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:55.358208 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa814824-6e18-40a3-b41b-9075474152e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:55.358547 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa814824-6e18-40a3-b41b-9075474152e5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:55.394051 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a4668e9-3779-43bb-890c-a2cf3f3bb4c6 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:55.428641 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a38d9d8-6474-470b-8727-f53d99ffaf25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:55.429153 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a38d9d8-6474-470b-8727-f53d99ffaf25 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:55.429471 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a38d9d8-6474-470b-8727-f53d99ffaf25 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:55.429741 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a38d9d8-6474-470b-8727-f53d99ffaf25 None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:55.430458 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a38d9d8-6474-470b-8727-f53d99ffaf25 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:55.430933 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a38d9d8-6474-470b-8727-f53d99ffaf25 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:55.432310 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a38d9d8-6474-470b-8727-f53d99ffaf25 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:55.433575 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa814824-6e18-40a3-b41b-9075474152e5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:55.434992 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa814824-6e18-40a3-b41b-9075474152e5 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:55.440168 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 230/447] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:41:55 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:55.446396 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a38d9d8-6474-470b-8727-f53d99ffaf25 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:55.537669 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 218/448] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:41:55 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 00:41:55.543297 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa814824-6e18-40a3-b41b-9075474152e5 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:55.551178 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:55.554768 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a38d9d8-6474-470b-8727-f53d99ffaf25 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:55.558766 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:55.596060 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30fedec-e551-41ee-a56d-60663d619f4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:55.596732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30fedec-e551-41ee-a56d-60663d619f4a None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:55.597016 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30fedec-e551-41ee-a56d-60663d619f4a None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:55.597315 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30fedec-e551-41ee-a56d-60663d619f4a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:55.662991 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa814824-6e18-40a3-b41b-9075474152e5 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:55.677102 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d30fedec-e551-41ee-a56d-60663d619f4a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:55.678793 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d30fedec-e551-41ee-a56d-60663d619f4a None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:55.707909 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1380a7b8-e9b6-45d6-907d-b8a2904792f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:55.708542 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1380a7b8-e9b6-45d6-907d-b8a2904792f4 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:55.708815 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1380a7b8-e9b6-45d6-907d-b8a2904792f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:55.709078 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1380a7b8-e9b6-45d6-907d-b8a2904792f4 None admin] PATH_INFO: `/v3/projects/c8e32a9668254c2e96354fd797883364` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:55.713293 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1380a7b8-e9b6-45d6-907d-b8a2904792f4 None admin] RBAC: Authorizing `identity:get_project(project_id=c8e32a9668254c2e96354fd797883364)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:55.713293 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1380a7b8-e9b6-45d6-907d-b8a2904792f4 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:55.714104 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 219/449] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:41:55 2022] GET /identity/v3/projects/c8e32a9668254c2e96354fd797883364 => generated 344 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:41:55.719549 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1380a7b8-e9b6-45d6-907d-b8a2904792f4 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:55.793302 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 231/450] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:41:55 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 344 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 25 00:41:55.795198 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30fedec-e551-41ee-a56d-60663d619f4a None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:55.799852 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:55.803467 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:55.836840 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1380a7b8-e9b6-45d6-907d-b8a2904792f4 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:55.876322 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d197a0a6-b1d5-4ad4-a06d-03b972a1bf7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:55.876950 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d197a0a6-b1d5-4ad4-a06d-03b972a1bf7b None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:55.877218 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d197a0a6-b1d5-4ad4-a06d-03b972a1bf7b None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:55.877526 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d197a0a6-b1d5-4ad4-a06d-03b972a1bf7b None admin] PATH_INFO: `/v3/groups/3593b329027d4ed7abe6cc428574877a` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:55.878621 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d197a0a6-b1d5-4ad4-a06d-03b972a1bf7b None admin] RBAC: Authorizing `identity:get_group(group_id=3593b329027d4ed7abe6cc428574877a)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:55.880061 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d197a0a6-b1d5-4ad4-a06d-03b972a1bf7b None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:55.881719 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 220/451] 10.222.0.52 () {56 vars in 1180 bytes} [Sun Sep 25 00:41:55 2022] GET /identity/v3/groups/3593b329027d4ed7abe6cc428574877a => generated 229 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:41:55.888514 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d197a0a6-b1d5-4ad4-a06d-03b972a1bf7b None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:55.910311 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30fedec-e551-41ee-a56d-60663d619f4a None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:55.944962 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d89c923-8763-4f21-8413-d991da31a64e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:55.945533 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d89c923-8763-4f21-8413-d991da31a64e None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:55.946330 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d89c923-8763-4f21-8413-d991da31a64e None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:55.946602 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d89c923-8763-4f21-8413-d991da31a64e None admin] PATH_INFO: `/v3/projects/86aea44da5fb4bdeb527c7ff257c5d43` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:55.947739 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d89c923-8763-4f21-8413-d991da31a64e None admin] RBAC: Authorizing `identity:get_project(project_id=86aea44da5fb4bdeb527c7ff257c5d43)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:55.949333 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d89c923-8763-4f21-8413-d991da31a64e None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:55.950977 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 232/452] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:41:55 2022] GET /identity/v3/projects/86aea44da5fb4bdeb527c7ff257c5d43 => generated 302 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:41:55.956568 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d89c923-8763-4f21-8413-d991da31a64e None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:55.994418 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d197a0a6-b1d5-4ad4-a06d-03b972a1bf7b None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:56.029148 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89517f82-5578-495d-baad-6f9e9288ea77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:56.029778 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89517f82-5578-495d-baad-6f9e9288ea77 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:56.030051 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89517f82-5578-495d-baad-6f9e9288ea77 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:56.030322 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89517f82-5578-495d-baad-6f9e9288ea77 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:56.030445 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:56.031041 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89517f82-5578-495d-baad-6f9e9288ea77 None admin] RBAC: Adding query filter params (group.id=3593b329027d4ed7abe6cc428574877a, role.id=2c242abf6ed8410ab0a4a354ca1794fb, scope.project.id=c8e32a9668254c2e96354fd797883364) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:56.031344 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89517f82-5578-495d-baad-6f9e9288ea77 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:56.032942 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89517f82-5578-495d-baad-6f9e9288ea77 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:56.039123 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 221/453] 10.222.0.52 () {56 vars in 1376 bytes} [Sun Sep 25 00:41:55 2022] GET /identity/v3/role_assignments?group.id=3593b329027d4ed7abe6cc428574877a&scope.project.id=c8e32a9668254c2e96354fd797883364&role.id=2c242abf6ed8410ab0a4a354ca1794fb => generated 611 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:41:56.064595 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d89c923-8763-4f21-8413-d991da31a64e None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:56.103278 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dbe827a-7517-47cc-911c-a0806b7cb60f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:56.103960 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbe827a-7517-47cc-911c-a0806b7cb60f None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:56.104233 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbe827a-7517-47cc-911c-a0806b7cb60f None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:56.104499 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbe827a-7517-47cc-911c-a0806b7cb60f None admin] PATH_INFO: `/v3/groups/f52544847fbb4d61b602bdeadf58d9d6` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:56.105814 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dbe827a-7517-47cc-911c-a0806b7cb60f None admin] RBAC: Authorizing `identity:get_group(group_id=f52544847fbb4d61b602bdeadf58d9d6)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:56.107295 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dbe827a-7517-47cc-911c-a0806b7cb60f None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:56.108994 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 233/454] 10.222.0.52 () {56 vars in 1180 bytes} [Sun Sep 25 00:41:55 2022] GET /identity/v3/groups/f52544847fbb4d61b602bdeadf58d9d6 => generated 226 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:41:56.115576 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89517f82-5578-495d-baad-6f9e9288ea77 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:56.125183 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:56.225925 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89517f82-5578-495d-baad-6f9e9288ea77 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:56.262384 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea8a77a5-410e-4aba-9d32-17a900a7e485 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:56.262992 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea8a77a5-410e-4aba-9d32-17a900a7e485 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:56.263273 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea8a77a5-410e-4aba-9d32-17a900a7e485 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:56.263534 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea8a77a5-410e-4aba-9d32-17a900a7e485 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:56.264258 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea8a77a5-410e-4aba-9d32-17a900a7e485 None admin] RBAC: Adding query filter params (group.id=f52544847fbb4d61b602bdeadf58d9d6, role.id=a917909d7aa84d00a841c77e7e7bdc8a, scope.project.id=86aea44da5fb4bdeb527c7ff257c5d43) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:56.264560 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea8a77a5-410e-4aba-9d32-17a900a7e485 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:56.266041 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea8a77a5-410e-4aba-9d32-17a900a7e485 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:56.272021 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 222/455] 10.222.0.52 () {56 vars in 1376 bytes} [Sun Sep 25 00:41:56 2022] GET /identity/v3/role_assignments?group.id=f52544847fbb4d61b602bdeadf58d9d6&scope.project.id=86aea44da5fb4bdeb527c7ff257c5d43&role.id=a917909d7aa84d00a841c77e7e7bdc8a => generated 611 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:58.967796 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-186cfd03-b3bb-43d6-a41b-aceb80346278 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:58.968285 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-186cfd03-b3bb-43d6-a41b-aceb80346278 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:58.968285 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-186cfd03-b3bb-43d6-a41b-aceb80346278 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:58.969862 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 234/456] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:58.974991 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d69b91-b433-428c-8450-c175c2eb01ad None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:58.975789 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d69b91-b433-428c-8450-c175c2eb01ad None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:58.976252 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d69b91-b433-428c-8450-c175c2eb01ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:59.002279 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-d8d69b91-b433-428c-8450-c175c2eb01ad None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:59.003817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-d8d69b91-b433-428c-8450-c175c2eb01ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:59.095158 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:59.099756 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 223/457] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:41:59.425541 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-186cfd03-b3bb-43d6-a41b-aceb80346278 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:59.460617 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d700b7-12d5-4c97-b196-c1a3b4efaff6 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:59.461835 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d700b7-12d5-4c97-b196-c1a3b4efaff6 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:59.462309 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d700b7-12d5-4c97-b196-c1a3b4efaff6 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:59.464234 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 224/458] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:59.469196 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbc8552-9d12-45f4-b800-3dc44e66ad48 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:59.469792 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbc8552-9d12-45f4-b800-3dc44e66ad48 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:59.470268 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbc8552-9d12-45f4-b800-3dc44e66ad48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:59.506256 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-7fbc8552-9d12-45f4-b800-3dc44e66ad48 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:59.507442 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-7fbc8552-9d12-45f4-b800-3dc44e66ad48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:59.547147 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-186cfd03-b3bb-43d6-a41b-aceb80346278 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:59.582209 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f6ba666-37e4-490b-a987-49fdb93cd56c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:59.583061 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6ba666-37e4-490b-a987-49fdb93cd56c None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:59.583503 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6ba666-37e4-490b-a987-49fdb93cd56c None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:59.583908 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6ba666-37e4-490b-a987-49fdb93cd56c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:59.593621 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:59.597796 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f6ba666-37e4-490b-a987-49fdb93cd56c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:59.599501 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f6ba666-37e4-490b-a987-49fdb93cd56c None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:59.619565 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-0f6ba666-37e4-490b-a987-49fdb93cd56c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:41:59.619565 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:41:59.619565 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:41:59.619565 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:41:59.619565 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:41:59.619565 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:41:59.619565 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:41:59.619565 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:41:59.619565 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:41:59.619565 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:41:59.619565 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:41:59.619565 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:41:59.619565 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:41:59.619565 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:41:59.619565 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:41:59.619565 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:41:59.620568 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:41:59.620568 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:41:59.620568 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:41:59.620568 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:41:59.620568 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:41:59.620568 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:41:59.620568 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:41:59.620568 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:41:59.620568 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:41:59.620568 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:41:59.620568 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:41:59.620568 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:41:59.620568 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:41:59.620568 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:41:59.620568 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:41:59.620568 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:41:59.620568 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:41:59.621450 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:41:59.621450 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:41:59.621450 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:41:59.623257 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 235/459] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:41:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:41:59.624286 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d05d3a24-eac9-4832-a04c-4919b301d523 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:59.624563 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d05d3a24-eac9-4832-a04c-4919b301d523 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:59.625534 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d05d3a24-eac9-4832-a04c-4919b301d523 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:59.627600 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 236/460] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:41:59.633499 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b424f7b6-1c2d-40c5-878a-7c81966bc70e None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:59.634442 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b424f7b6-1c2d-40c5-878a-7c81966bc70e None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:59.635310 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b424f7b6-1c2d-40c5-878a-7c81966bc70e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:59.656302 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 225/461] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:41:59.656803 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-b424f7b6-1c2d-40c5-878a-7c81966bc70e None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:59.657749 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fbc8552-9d12-45f4-b800-3dc44e66ad48 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:41:59.658229 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-b424f7b6-1c2d-40c5-878a-7c81966bc70e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:59.799748 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 237/462] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 25 00:41:59.827095 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fbc8552-9d12-45f4-b800-3dc44e66ad48 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:41:59.840037 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76735595-1421-4ef2-85a5-1ecb5ce0f0d4 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:59.840636 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76735595-1421-4ef2-85a5-1ecb5ce0f0d4 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:59.841491 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76735595-1421-4ef2-85a5-1ecb5ce0f0d4 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:59.843625 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 238/463] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:41:59 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 00:41:59.848498 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4645f6-478e-4426-b221-8304ba51e708 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:59.848998 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4645f6-478e-4426-b221-8304ba51e708 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:59.849543 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4645f6-478e-4426-b221-8304ba51e708 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:59.869226 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:41:59.880643 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f99886b-9067-45fa-9e99-cdc8251683b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:41:59.881248 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f99886b-9067-45fa-9e99-cdc8251683b2 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:41:59.882671 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f99886b-9067-45fa-9e99-cdc8251683b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:41:59.882965 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f99886b-9067-45fa-9e99-cdc8251683b2 None admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:41:59.885236 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f99886b-9067-45fa-9e99-cdc8251683b2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:41:59.885562 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f99886b-9067-45fa-9e99-cdc8251683b2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:41:59.887929 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f99886b-9067-45fa-9e99-cdc8251683b2 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:41:59.889272 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-4a4645f6-478e-4426-b221-8304ba51e708 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:41:59.891879 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-4a4645f6-478e-4426-b221-8304ba51e708 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:41:59.912203 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 226/464] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:41:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:41:59.917691 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f99886b-9067-45fa-9e99-cdc8251683b2 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:00.017472 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 239/465] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:41:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 25 00:42:00.018996 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a4645f6-478e-4426-b221-8304ba51e708 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:00.059949 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f99886b-9067-45fa-9e99-cdc8251683b2 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:00.113871 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:00.120452 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a34041c0-33a1-4da6-b6c0-5103ba1ec37c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:00.121396 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a34041c0-33a1-4da6-b6c0-5103ba1ec37c None admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:00.121977 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a34041c0-33a1-4da6-b6c0-5103ba1ec37c None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:00.122447 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a34041c0-33a1-4da6-b6c0-5103ba1ec37c None admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:00.123311 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a34041c0-33a1-4da6-b6c0-5103ba1ec37c None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:00.124969 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a34041c0-33a1-4da6-b6c0-5103ba1ec37c None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:00.156020 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 227/466] 10.222.0.52 () {60 vars in 1132 bytes} [Sun Sep 25 00:41:59 2022] POST /identity/v3/users => generated 246 bytes in 234 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:42:00.157533 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a34041c0-33a1-4da6-b6c0-5103ba1ec37c None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:00.167264 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a4645f6-478e-4426-b221-8304ba51e708 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:00.205337 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70759493-b6fc-4486-9574-73e6ce616a4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:00.206044 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70759493-b6fc-4486-9574-73e6ce616a4f None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:00.206228 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70759493-b6fc-4486-9574-73e6ce616a4f None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:00.206496 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70759493-b6fc-4486-9574-73e6ce616a4f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:00.220628 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70759493-b6fc-4486-9574-73e6ce616a4f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:00.220628 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70759493-b6fc-4486-9574-73e6ce616a4f None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:00.235941 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-70759493-b6fc-4486-9574-73e6ce616a4f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:00.235941 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:00.235941 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:00.235941 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:00.235941 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:00.235941 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:00.235941 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:00.235941 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:00.235941 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:00.235941 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:00.235941 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:00.235941 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:00.235941 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:00.235941 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:00.235941 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:00.235941 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:00.236934 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:00.236934 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:00.236934 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:00.236934 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:00.236934 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:00.236934 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:00.236934 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:00.236934 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:00.236934 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:00.236934 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:00.236934 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:00.236934 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:00.236934 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:00.236934 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:00.236934 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:00.236934 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:00.236934 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:00.238785 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:00.238785 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:00.238785 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:00.238785 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 240/467] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:00.246267 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70759493-b6fc-4486-9574-73e6ce616a4f None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:00.251573 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:00.345890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a34041c0-33a1-4da6-b6c0-5103ba1ec37c None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:00.381363 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70759493-b6fc-4486-9574-73e6ce616a4f None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:00.385793 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:00.389480 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e5f260-c1e8-472e-884f-60dddccfb131 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:00.390216 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e5f260-c1e8-472e-884f-60dddccfb131 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:00.390560 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e5f260-c1e8-472e-884f-60dddccfb131 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:00.390896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e5f260-c1e8-472e-884f-60dddccfb131 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:00.404323 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e5f260-c1e8-472e-884f-60dddccfb131 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:00.406382 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e5f260-c1e8-472e-884f-60dddccfb131 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:00.423901 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-61e5f260-c1e8-472e-884f-60dddccfb131 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:00.423901 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:00.423901 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:00.423901 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:00.423901 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:00.423901 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:00.423901 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:00.423901 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:00.423901 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:00.423901 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:00.423901 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:00.423901 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:00.423901 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:00.423901 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:00.423901 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:00.423901 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:00.424876 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:00.424876 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:00.424876 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:00.424876 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:00.424876 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:00.424876 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:00.424876 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:00.424876 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:00.424876 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:00.424876 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:00.424876 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:00.424876 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:00.424876 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:00.424876 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:00.424876 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:00.424876 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:00.424876 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:00.425856 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:00.425856 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:00.425856 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:00.425856 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 228/468] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 268 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:00.425856 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e5f260-c1e8-472e-884f-60dddccfb131 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:00.429582 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-accc44fc-6c46-468c-89f4-c89bccaa4f32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:00.430433 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-accc44fc-6c46-468c-89f4-c89bccaa4f32 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:00.430761 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-accc44fc-6c46-468c-89f4-c89bccaa4f32 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:00.431107 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-accc44fc-6c46-468c-89f4-c89bccaa4f32 None admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:00.432372 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-accc44fc-6c46-468c-89f4-c89bccaa4f32 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:00.432372 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-accc44fc-6c46-468c-89f4-c89bccaa4f32 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:00.434280 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-accc44fc-6c46-468c-89f4-c89bccaa4f32 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:00.448490 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 241/469] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:00.450319 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-accc44fc-6c46-468c-89f4-c89bccaa4f32 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:00.548410 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:00.552805 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e5f260-c1e8-472e-884f-60dddccfb131 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:00.589963 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-accc44fc-6c46-468c-89f4-c89bccaa4f32 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:00.604612 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52edee90-2c9e-4c42-8370-45ade381816a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:00.605169 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52edee90-2c9e-4c42-8370-45ade381816a None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:00.605484 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52edee90-2c9e-4c42-8370-45ade381816a None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:00.605756 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52edee90-2c9e-4c42-8370-45ade381816a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:00.621238 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52edee90-2c9e-4c42-8370-45ade381816a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:00.623463 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52edee90-2c9e-4c42-8370-45ade381816a None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:00.633754 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-600a03d1-1630-4b9a-b7ca-9a961491a4f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:00.634374 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600a03d1-1630-4b9a-b7ca-9a961491a4f6 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:00.634672 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600a03d1-1630-4b9a-b7ca-9a961491a4f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:00.635469 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600a03d1-1630-4b9a-b7ca-9a961491a4f6 None admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:00.636161 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-600a03d1-1630-4b9a-b7ca-9a961491a4f6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:00.636598 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-600a03d1-1630-4b9a-b7ca-9a961491a4f6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:00.638158 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-600a03d1-1630-4b9a-b7ca-9a961491a4f6 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:00.642340 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-52edee90-2c9e-4c42-8370-45ade381816a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:00.642340 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:00.642340 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:00.642340 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:00.642340 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:00.642340 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:00.642340 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:00.642340 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:00.642340 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:00.642340 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:00.642340 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:00.642340 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:00.642340 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:00.642340 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:00.642340 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:00.642340 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:00.643222 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:00.643222 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:00.643222 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:00.643222 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:00.643222 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:00.643222 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:00.643222 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:00.643222 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:00.643222 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:00.643222 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:00.643222 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:00.643222 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:00.643222 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:00.643222 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:00.643222 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:00.643222 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:00.643222 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:00.644065 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:00.644065 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:00.644065 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:00.646204 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 229/470] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:00.652288 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52edee90-2c9e-4c42-8370-45ade381816a None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:00.652421 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 242/471] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:00.653329 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-600a03d1-1630-4b9a-b7ca-9a961491a4f6 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:00.682088 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:00.783259 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-600a03d1-1630-4b9a-b7ca-9a961491a4f6 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:00.810301 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52edee90-2c9e-4c42-8370-45ade381816a None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:00.824385 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2516743-c118-43e6-8ef5-6b6738d32edb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:00.825000 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2516743-c118-43e6-8ef5-6b6738d32edb None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:00.825298 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2516743-c118-43e6-8ef5-6b6738d32edb None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:00.825576 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2516743-c118-43e6-8ef5-6b6738d32edb None admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:00.826222 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2516743-c118-43e6-8ef5-6b6738d32edb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:00.826522 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2516743-c118-43e6-8ef5-6b6738d32edb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:00.828670 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2516743-c118-43e6-8ef5-6b6738d32edb None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:00.849289 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 243/472] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:00.849289 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2516743-c118-43e6-8ef5-6b6738d32edb None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:00.854387 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38191850-200b-416f-a266-3113aca40f7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:00.855343 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38191850-200b-416f-a266-3113aca40f7c None admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:00.855862 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38191850-200b-416f-a266-3113aca40f7c None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:00.856400 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38191850-200b-416f-a266-3113aca40f7c None admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:00.857234 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38191850-200b-416f-a266-3113aca40f7c None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:00.858944 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38191850-200b-416f-a266-3113aca40f7c None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:00.881884 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 230/473] 10.222.0.52 () {60 vars in 1132 bytes} [Sun Sep 25 00:42:00 2022] POST /identity/v3/users => generated 248 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:42:00.882687 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38191850-200b-416f-a266-3113aca40f7c None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:00.901301 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:00.944689 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:00.964819 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2516743-c118-43e6-8ef5-6b6738d32edb None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:01.003293 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e990c851-0fa8-491c-9f0d-fe1cf2dd4cf1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:01.004002 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e990c851-0fa8-491c-9f0d-fe1cf2dd4cf1 None admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:01.004348 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e990c851-0fa8-491c-9f0d-fe1cf2dd4cf1 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:01.004686 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e990c851-0fa8-491c-9f0d-fe1cf2dd4cf1 None admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:01.005494 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e990c851-0fa8-491c-9f0d-fe1cf2dd4cf1 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:01.007040 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e990c851-0fa8-491c-9f0d-fe1cf2dd4cf1 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:01.008335 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38191850-200b-416f-a266-3113aca40f7c None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:01.036474 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 244/474] 10.222.0.52 () {60 vars in 1132 bytes} [Sun Sep 25 00:42:00 2022] POST /identity/v3/users => generated 249 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:42:01.038004 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093c0a93-d30e-444a-b028-7e3a3bd0140d None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:01.038361 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093c0a93-d30e-444a-b028-7e3a3bd0140d None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:01.038697 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093c0a93-d30e-444a-b028-7e3a3bd0140d None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:01.045874 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 245/475] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:42:01 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:42:01.047584 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-349da532-0343-44c1-a2ad-17a39279f1b1 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:01.048006 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-349da532-0343-44c1-a2ad-17a39279f1b1 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:01.048283 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-349da532-0343-44c1-a2ad-17a39279f1b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:01.069706 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-349da532-0343-44c1-a2ad-17a39279f1b1 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:42:01.071575 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-349da532-0343-44c1-a2ad-17a39279f1b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:42:01.072858 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc0d004-64fe-408b-8f8b-73de0d7bd9ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:01.073523 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc0d004-64fe-408b-8f8b-73de0d7bd9ff None admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:01.073803 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc0d004-64fe-408b-8f8b-73de0d7bd9ff None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:01.074076 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc0d004-64fe-408b-8f8b-73de0d7bd9ff None admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:01.074733 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc0d004-64fe-408b-8f8b-73de0d7bd9ff None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:01.076184 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc0d004-64fe-408b-8f8b-73de0d7bd9ff None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:01.117043 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 231/476] 10.222.0.52 () {60 vars in 1132 bytes} [Sun Sep 25 00:42:00 2022] POST /identity/v3/users => generated 248 bytes in 234 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:42:01.157131 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 246/477] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:42:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:42:01.441578 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc0d004-64fe-408b-8f8b-73de0d7bd9ff None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:01.514556 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:01.561000 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc0d004-64fe-408b-8f8b-73de0d7bd9ff None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:01.597080 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5e5845-d111-403d-86a8-15784d831c90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:01.597719 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5e5845-d111-403d-86a8-15784d831c90 None admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:01.597990 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5e5845-d111-403d-86a8-15784d831c90 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:01.598255 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5e5845-d111-403d-86a8-15784d831c90 None admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:01.598928 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c5e5845-d111-403d-86a8-15784d831c90 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:01.600296 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c5e5845-d111-403d-86a8-15784d831c90 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:01.608806 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.sql.core [None req-6c5e5845-d111-403d-86a8-15784d831c90 None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 25 00:42:01.608806 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Sep 25 00:42:01.608806 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [parameters: {'id': 'c8664208c25e4593bdcddfd0a847945c', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Sep 25 00:42:01.608806 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81387) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 25 00:42:01.626624 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-6c5e5845-d111-403d-86a8-15784d831c90 None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Sep 25 00:42:01.626624 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:01.626624 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 25 00:42:01.626624 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 25 00:42:01.626624 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 25 00:42:01.626624 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 25 00:42:01.626624 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 25 00:42:01.626624 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application result = self._query(query) Sep 25 00:42:01.626624 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 25 00:42:01.626624 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application conn.query(q) Sep 25 00:42:01.626624 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 25 00:42:01.626624 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 25 00:42:01.626624 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 25 00:42:01.626624 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application result.read() Sep 25 00:42:01.626624 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 25 00:42:01.627566 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 25 00:42:01.627566 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 25 00:42:01.627566 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 25 00:42:01.627566 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 25 00:42:01.627566 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 25 00:42:01.627566 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 25 00:42:01.627566 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 25 00:42:01.627566 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 25 00:42:01.627566 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:01.627566 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 25 00:42:01.627566 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:01.627566 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:01.627566 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 25 00:42:01.627566 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 25 00:42:01.627566 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Sep 25 00:42:01.627566 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return ref.to_dict() Sep 25 00:42:01.627566 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 25 00:42:01.627566 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application next(self.gen) Sep 25 00:42:01.628605 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 25 00:42:01.628605 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application yield resource Sep 25 00:42:01.628605 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 25 00:42:01.628605 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application next(self.gen) Sep 25 00:42:01.628605 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 25 00:42:01.628605 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application self.session.rollback() Sep 25 00:42:01.628605 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 25 00:42:01.628605 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application self.force_reraise() Sep 25 00:42:01.628605 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 25 00:42:01.628605 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise self.value Sep 25 00:42:01.628605 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 25 00:42:01.628605 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 25 00:42:01.628605 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 25 00:42:01.628605 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application session.commit() Sep 25 00:42:01.628605 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 25 00:42:01.628605 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 25 00:42:01.628605 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 25 00:42:01.629535 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application self._prepare_impl() Sep 25 00:42:01.629535 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 25 00:42:01.629535 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application self.session.flush() Sep 25 00:42:01.629535 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 25 00:42:01.629535 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application self._flush(objects) Sep 25 00:42:01.629535 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 25 00:42:01.629535 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 25 00:42:01.629535 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 25 00:42:01.629535 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application compat.raise_( Sep 25 00:42:01.629535 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 25 00:42:01.629535 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception Sep 25 00:42:01.629535 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 25 00:42:01.629535 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application flush_context.execute() Sep 25 00:42:01.629535 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 25 00:42:01.629535 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rec.execute(self) Sep 25 00:42:01.629535 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 25 00:42:01.629535 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 25 00:42:01.630423 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 25 00:42:01.630423 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 25 00:42:01.630423 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Sep 25 00:42:01.630423 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application c = connection._execute_20( Sep 25 00:42:01.630423 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 25 00:42:01.630423 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 25 00:42:01.630423 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 25 00:42:01.630423 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 25 00:42:01.630423 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 25 00:42:01.630423 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 25 00:42:01.630423 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 25 00:42:01.630423 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 25 00:42:01.630423 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 25 00:42:01.630423 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 25 00:42:01.630423 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 25 00:42:01.630423 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception Sep 25 00:42:01.631275 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 25 00:42:01.631275 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 25 00:42:01.631275 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 25 00:42:01.631275 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 25 00:42:01.631275 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 25 00:42:01.631275 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application result = self._query(query) Sep 25 00:42:01.631275 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 25 00:42:01.631275 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application conn.query(q) Sep 25 00:42:01.631275 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 25 00:42:01.631275 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 25 00:42:01.631275 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 25 00:42:01.631275 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application result.read() Sep 25 00:42:01.631275 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 25 00:42:01.631275 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 25 00:42:01.631275 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 25 00:42:01.631275 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 25 00:42:01.632069 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 25 00:42:01.632069 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 25 00:42:01.632069 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 25 00:42:01.632069 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 25 00:42:01.632069 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 25 00:42:01.632069 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Sep 25 00:42:01.632069 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [parameters: {'id': 'c8664208c25e4593bdcddfd0a847945c', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Sep 25 00:42:01.632069 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 25 00:42:01.632069 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:01.632069 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 00:42:01.632069 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:01.632069 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:01.632069 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:01.632069 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:01.632069 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:01.632069 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:01.632905 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:01.632905 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:01.632905 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:01.632905 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:01.632905 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:01.632905 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:01.632905 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Sep 25 00:42:01.632905 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Sep 25 00:42:01.632905 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:01.632905 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:01.632905 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Sep 25 00:42:01.632905 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Sep 25 00:42:01.632905 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 25 00:42:01.632905 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 25 00:42:01.632905 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Sep 25 00:42:01.632905 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:01.637996 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 232/478] 10.222.0.52 () {60 vars in 1131 bytes} [Sun Sep 25 00:42:01 2022] POST /identity/v3/roles => generated 144 bytes in 187 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:01.638149 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-349da532-0343-44c1-a2ad-17a39279f1b1 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:01.708679 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:01.741529 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-349da532-0343-44c1-a2ad-17a39279f1b1 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:01.779202 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e941b70-a268-4583-9a23-eacf21d42ad3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:01.779828 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e941b70-a268-4583-9a23-eacf21d42ad3 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:01.780103 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e941b70-a268-4583-9a23-eacf21d42ad3 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:01.780365 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e941b70-a268-4583-9a23-eacf21d42ad3 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:01.790993 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e941b70-a268-4583-9a23-eacf21d42ad3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:42:01.795854 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-0e941b70-a268-4583-9a23-eacf21d42ad3 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 00:42:01.795854 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:01.795854 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:01.795854 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:01.795854 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:01.795854 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:01.795854 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:01.795854 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:01.795854 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:01.795854 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:01.795854 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:01.795854 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:01.795854 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:42:01.795854 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:42:01.795854 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:42:01.795854 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:42:01.796757 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:42:01.796757 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:42:01.796757 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:42:01.796757 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:42:01.796757 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:01.796757 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:01.796757 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:01.796757 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:01.796757 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:01.796757 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:01.796757 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:01.796757 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:01.796757 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:01.796757 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:01.796757 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:01.796757 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:01.797619 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:01.797619 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:01.797619 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:01.797619 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:01.797619 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:42:01.797619 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:42:01.797619 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:42:01.797619 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:42:01.797619 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:42:01.797619 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:42:01.797619 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 00:42:01.797619 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:01.797619 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 247/479] 10.222.0.52 () {56 vars in 1114 bytes} [Sun Sep 25 00:42:01 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:01.803113 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5e5845-d111-403d-86a8-15784d831c90 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:01.935408 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5e5845-d111-403d-86a8-15784d831c90 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:01.963589 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:01.973595 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b274bba9-1429-4e14-9584-0d3742940565 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:01.974200 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b274bba9-1429-4e14-9584-0d3742940565 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:01.974476 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b274bba9-1429-4e14-9584-0d3742940565 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:01.974740 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b274bba9-1429-4e14-9584-0d3742940565 None admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:01.975464 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b274bba9-1429-4e14-9584-0d3742940565 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:01.975766 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b274bba9-1429-4e14-9584-0d3742940565 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:01.977906 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b274bba9-1429-4e14-9584-0d3742940565 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:01.987142 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 233/480] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:42:01 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:02.926568 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086e9553-60df-4f16-b6b3-062b9e6ad632 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:02.927060 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086e9553-60df-4f16-b6b3-062b9e6ad632 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:02.927060 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086e9553-60df-4f16-b6b3-062b9e6ad632 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:02.928605 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 248/481] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:42:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:42:02.934078 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24c7925-c80a-403f-a5b0-d4a6011508db None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:02.934512 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24c7925-c80a-403f-a5b0-d4a6011508db None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:02.934690 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24c7925-c80a-403f-a5b0-d4a6011508db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:02.962819 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-c24c7925-c80a-403f-a5b0-d4a6011508db None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:42:02.964258 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-c24c7925-c80a-403f-a5b0-d4a6011508db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:42:03.055444 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 234/482] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:42:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:42:03.388361 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-086e9553-60df-4f16-b6b3-062b9e6ad632 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:03.519653 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcf9b0d-6207-4f34-81f1-b14e5ea4981a None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:03.520348 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcf9b0d-6207-4f34-81f1-b14e5ea4981a None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:03.520817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcf9b0d-6207-4f34-81f1-b14e5ea4981a None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:03.522387 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 235/483] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:42:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 00:42:03.527990 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d9e6d8-ec2c-494e-97da-27ff4bc293f8 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:03.528249 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d9e6d8-ec2c-494e-97da-27ff4bc293f8 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:03.528526 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d9e6d8-ec2c-494e-97da-27ff4bc293f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:03.530208 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-086e9553-60df-4f16-b6b3-062b9e6ad632 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:03.567767 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-71d9e6d8-ec2c-494e-97da-27ff4bc293f8 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:42:03.568937 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-71d9e6d8-ec2c-494e-97da-27ff4bc293f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:42:03.580426 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:03.589365 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72583004-6d1e-4096-bee2-2dc933e319a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:03.590004 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72583004-6d1e-4096-bee2-2dc933e319a4 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:03.591125 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72583004-6d1e-4096-bee2-2dc933e319a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:03.591404 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72583004-6d1e-4096-bee2-2dc933e319a4 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:03.614201 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72583004-6d1e-4096-bee2-2dc933e319a4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:42:03.619174 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-72583004-6d1e-4096-bee2-2dc933e319a4 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 25 00:42:03.619174 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:03.619174 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:03.619174 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:03.619174 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:03.619174 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:03.619174 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:03.619174 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:03.619174 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:03.619174 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:03.619174 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:03.619174 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:03.619174 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:42:03.619174 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:42:03.619174 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:42:03.619174 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:42:03.620095 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:42:03.620095 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:42:03.620095 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:42:03.620095 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:42:03.620095 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:03.620095 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:03.620095 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:03.620095 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:03.620095 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:03.620095 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:03.620095 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:03.620095 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:03.620095 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:03.620095 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:03.620095 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:03.620095 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:03.620986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:03.620986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:03.620986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:03.620986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:03.620986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:42:03.620986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:42:03.620986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:42:03.620986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:42:03.620986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:42:03.620986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:42:03.620986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 25 00:42:03.620986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:03.621659 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 249/484] 10.222.0.52 () {56 vars in 1102 bytes} [Sun Sep 25 00:42:03 2022] GET /identity/v3/roles/service => generated 85 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:03.628116 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72583004-6d1e-4096-bee2-2dc933e319a4 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:03.705502 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 236/485] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:42:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:42:03.774451 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2271ed-9d96-4971-8cf4-23b637381699 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:03.774784 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2271ed-9d96-4971-8cf4-23b637381699 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:03.775115 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2271ed-9d96-4971-8cf4-23b637381699 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:03.776969 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 237/486] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:42:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 00:42:03.782314 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e21002b-043d-437a-820e-3964d43b0551 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:03.783241 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e21002b-043d-437a-820e-3964d43b0551 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:03.783737 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e21002b-043d-437a-820e-3964d43b0551 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:03.805890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-8e21002b-043d-437a-820e-3964d43b0551 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:42:03.807243 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-8e21002b-043d-437a-820e-3964d43b0551 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:42:03.866281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72583004-6d1e-4096-bee2-2dc933e319a4 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:03.893627 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:03.936268 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 238/487] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:42:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:42:03.940531 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd56e7c-75bf-4934-9858-e2fa131f14e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:03.941222 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd56e7c-75bf-4934-9858-e2fa131f14e8 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:03.941597 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd56e7c-75bf-4934-9858-e2fa131f14e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:03.941975 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd56e7c-75bf-4934-9858-e2fa131f14e8 None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:03.942711 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cd56e7c-75bf-4934-9858-e2fa131f14e8 None admin] RBAC: Adding query filter params (name=service) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:03.943079 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cd56e7c-75bf-4934-9858-e2fa131f14e8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:03.944517 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cd56e7c-75bf-4934-9858-e2fa131f14e8 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:03.954544 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 250/488] 10.222.0.52 () {56 vars in 1103 bytes} [Sun Sep 25 00:42:03 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 327 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:42:03.959753 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e21002b-043d-437a-820e-3964d43b0551 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:04.070496 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e21002b-043d-437a-820e-3964d43b0551 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:04.107542 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd56e7c-75bf-4934-9858-e2fa131f14e8 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:04.116322 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad9c483-1ef4-49d1-8b3b-348852d8157d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:04.116956 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad9c483-1ef4-49d1-8b3b-348852d8157d None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:04.117303 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad9c483-1ef4-49d1-8b3b-348852d8157d None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:04.117570 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad9c483-1ef4-49d1-8b3b-348852d8157d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:04.136725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aad9c483-1ef4-49d1-8b3b-348852d8157d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:04.140936 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aad9c483-1ef4-49d1-8b3b-348852d8157d None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:04.170653 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-aad9c483-1ef4-49d1-8b3b-348852d8157d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:04.170653 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:04.170653 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:04.170653 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:04.170653 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:04.170653 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:04.170653 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:04.170653 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:04.170653 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:04.170653 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:04.170653 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:04.170653 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:04.170653 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:04.170653 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:04.170653 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:04.170653 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:04.171619 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:04.171619 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:04.171619 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:04.171619 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:04.171619 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:04.171619 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:04.171619 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:04.171619 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:04.171619 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:04.171619 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:04.171619 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:04.171619 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:04.171619 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:04.171619 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:04.171619 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:04.171619 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:04.171619 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:04.172483 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:04.172483 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:04.172483 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:04.174267 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 239/489] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:42:04.182185 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad9c483-1ef4-49d1-8b3b-348852d8157d None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:04.191305 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:04.244496 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd56e7c-75bf-4934-9858-e2fa131f14e8 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:04.245625 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:04.285948 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef090c09-cc02-4962-a7eb-e5bc3cb44dfa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:04.286954 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef090c09-cc02-4962-a7eb-e5bc3cb44dfa None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:04.287430 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef090c09-cc02-4962-a7eb-e5bc3cb44dfa None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:04.287925 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef090c09-cc02-4962-a7eb-e5bc3cb44dfa None admin] PATH_INFO: `/v3/roles/service` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:04.302392 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef090c09-cc02-4962-a7eb-e5bc3cb44dfa None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:42:04.309499 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-ef090c09-cc02-4962-a7eb-e5bc3cb44dfa None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 25 00:42:04.309499 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:04.309499 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:04.309499 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:04.309499 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:04.309499 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:04.309499 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:04.309499 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:04.309499 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:04.309499 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:04.309499 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:04.309499 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:04.309499 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:42:04.309499 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:42:04.309499 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:42:04.309499 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:42:04.310437 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:42:04.310437 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:42:04.310437 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:42:04.310437 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:42:04.310437 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:04.310437 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:04.310437 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:04.310437 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:04.310437 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:04.310437 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:04.310437 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:04.310437 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:04.310437 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:04.310437 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:04.310437 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:04.310437 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:04.311248 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:04.311248 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:04.311248 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:04.311248 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:04.311248 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:42:04.311248 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:42:04.311248 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:42:04.311248 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:42:04.311248 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:42:04.311248 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:42:04.311248 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 25 00:42:04.311248 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:04.314226 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 251/490] 10.222.0.52 () {56 vars in 1102 bytes} [Sun Sep 25 00:42:04 2022] GET /identity/v3/roles/service => generated 85 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:04.315265 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef090c09-cc02-4962-a7eb-e5bc3cb44dfa None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:04.346983 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad9c483-1ef4-49d1-8b3b-348852d8157d None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:04.386260 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e767f1-f7e8-453a-b518-c6770f049cd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:04.386926 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e767f1-f7e8-453a-b518-c6770f049cd2 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:04.387276 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e767f1-f7e8-453a-b518-c6770f049cd2 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:04.387472 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e767f1-f7e8-453a-b518-c6770f049cd2 None admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:04.388163 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25e767f1-f7e8-453a-b518-c6770f049cd2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:04.388469 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25e767f1-f7e8-453a-b518-c6770f049cd2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:04.390175 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25e767f1-f7e8-453a-b518-c6770f049cd2 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:04.403889 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 240/491] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:04.404564 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e767f1-f7e8-453a-b518-c6770f049cd2 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:04.433507 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef090c09-cc02-4962-a7eb-e5bc3cb44dfa None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:04.447570 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:04.470504 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fddcc489-d682-49f8-9060-0bf93b8aa784 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:04.471129 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddcc489-d682-49f8-9060-0bf93b8aa784 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:04.471414 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddcc489-d682-49f8-9060-0bf93b8aa784 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:04.471685 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddcc489-d682-49f8-9060-0bf93b8aa784 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:04.487131 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fddcc489-d682-49f8-9060-0bf93b8aa784 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:42:04.493245 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-fddcc489-d682-49f8-9060-0bf93b8aa784 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 25 00:42:04.493245 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:04.493245 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:04.493245 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:04.493245 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:04.493245 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:04.493245 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:04.493245 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:04.493245 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:04.493245 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:04.493245 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:04.493245 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:04.493245 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:42:04.493245 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:42:04.493245 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:42:04.493245 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:42:04.494165 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:42:04.494165 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:42:04.494165 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:42:04.494165 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:42:04.494165 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:04.494165 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:04.494165 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:04.494165 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:04.494165 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:04.494165 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:04.494165 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:04.494165 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:04.494165 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:04.494165 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:04.494165 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:04.494165 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:04.495056 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:04.495056 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:04.495056 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:04.495056 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:04.495056 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:42:04.495056 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:42:04.495056 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:42:04.495056 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:42:04.495056 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:42:04.495056 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:42:04.495056 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 25 00:42:04.495056 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:04.495056 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 252/492] 10.222.0.52 () {56 vars in 1102 bytes} [Sun Sep 25 00:42:04 2022] GET /identity/v3/roles/service => generated 85 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:04.499265 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345fa8c2-a65f-432c-bc1d-ee72219bd333 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:04.499265 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345fa8c2-a65f-432c-bc1d-ee72219bd333 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:04.499265 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345fa8c2-a65f-432c-bc1d-ee72219bd333 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:04.501958 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 253/493] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:42:04 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:42:04.502771 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-345fa8c2-a65f-432c-bc1d-ee72219bd333 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:04.515306 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:04.526908 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e767f1-f7e8-453a-b518-c6770f049cd2 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:04.564783 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0310157c-3eea-49ab-9429-a579073547a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:04.565420 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0310157c-3eea-49ab-9429-a579073547a6 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:04.565694 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0310157c-3eea-49ab-9429-a579073547a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:04.565962 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0310157c-3eea-49ab-9429-a579073547a6 None admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:04.566611 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0310157c-3eea-49ab-9429-a579073547a6 None admin] RBAC: Adding query filter params (name=service) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:04.566905 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0310157c-3eea-49ab-9429-a579073547a6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:04.568267 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0310157c-3eea-49ab-9429-a579073547a6 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:04.577572 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 241/494] 10.222.0.52 () {56 vars in 1103 bytes} [Sun Sep 25 00:42:04 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:04.578362 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0310157c-3eea-49ab-9429-a579073547a6 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:04.622499 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-345fa8c2-a65f-432c-bc1d-ee72219bd333 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:04.666698 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbaaac3a-023f-4db0-b59b-ecddc136cb9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:04.667329 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbaaac3a-023f-4db0-b59b-ecddc136cb9e None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:04.667598 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbaaac3a-023f-4db0-b59b-ecddc136cb9e None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:04.667979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbaaac3a-023f-4db0-b59b-ecddc136cb9e None admin] PATH_INFO: `/v3/users/nova` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:04.676950 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbaaac3a-023f-4db0-b59b-ecddc136cb9e None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:04.678664 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbaaac3a-023f-4db0-b59b-ecddc136cb9e None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:04.687183 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0310157c-3eea-49ab-9429-a579073547a6 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:04.694974 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-cbaaac3a-023f-4db0-b59b-ecddc136cb9e None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 25 00:42:04.694974 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:04.694974 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:04.694974 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:04.694974 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:04.694974 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:04.694974 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:04.694974 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:04.694974 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:04.694974 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:04.694974 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:04.694974 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:04.694974 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 00:42:04.694974 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 00:42:04.694974 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 00:42:04.694974 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 00:42:04.696438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:04.696438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:04.696438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 00:42:04.696438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:04.696438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 00:42:04.696438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:04.696438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:04.696438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:04.696438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:04.696438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:04.696438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:04.696438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:04.696438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:04.696438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:04.696438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:04.696438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:04.696438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:04.696438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:04.697389 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:04.697389 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:04.697389 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 00:42:04.697389 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 00:42:04.697389 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 00:42:04.697389 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 00:42:04.697389 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 00:42:04.697389 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 00:42:04.697389 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 25 00:42:04.697389 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:04.698543 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 254/495] 10.222.0.52 () {56 vars in 1093 bytes} [Sun Sep 25 00:42:04 2022] GET /identity/v3/users/nova => generated 82 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:04.698694 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c0b7e0-5b0a-40b6-b2f2-b7c80f47fbee None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:04.698946 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c0b7e0-5b0a-40b6-b2f2-b7c80f47fbee None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:04.699214 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c0b7e0-5b0a-40b6-b2f2-b7c80f47fbee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:04.719629 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-75c0b7e0-5b0a-40b6-b2f2-b7c80f47fbee None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:42:04.720609 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-75c0b7e0-5b0a-40b6-b2f2-b7c80f47fbee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:42:04.734781 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bcceebe-bab5-48bf-aa9f-a115db7532ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:04.735443 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bcceebe-bab5-48bf-aa9f-a115db7532ec None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:04.735728 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bcceebe-bab5-48bf-aa9f-a115db7532ec None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:04.736063 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bcceebe-bab5-48bf-aa9f-a115db7532ec None admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:04.736773 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bcceebe-bab5-48bf-aa9f-a115db7532ec None admin] RBAC: Adding query filter params (name=service) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:04.737005 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bcceebe-bab5-48bf-aa9f-a115db7532ec None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:04.738731 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bcceebe-bab5-48bf-aa9f-a115db7532ec None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:04.748024 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 242/496] 10.222.0.52 () {56 vars in 1103 bytes} [Sun Sep 25 00:42:04 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:04.748519 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bcceebe-bab5-48bf-aa9f-a115db7532ec None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:04.754381 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:04.768046 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:04.804641 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 255/497] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:42:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:42:04.805882 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75c0b7e0-5b0a-40b6-b2f2-b7c80f47fbee None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:04.894600 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bcceebe-bab5-48bf-aa9f-a115db7532ec None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:04.935487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-159c7308-ef44-4bfc-900b-9ec2e8ee9417 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:04.936124 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159c7308-ef44-4bfc-900b-9ec2e8ee9417 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:04.936406 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159c7308-ef44-4bfc-900b-9ec2e8ee9417 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:04.937025 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75c0b7e0-5b0a-40b6-b2f2-b7c80f47fbee None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:04.937144 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159c7308-ef44-4bfc-900b-9ec2e8ee9417 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:04.949013 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-159c7308-ef44-4bfc-900b-9ec2e8ee9417 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:04.950550 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-159c7308-ef44-4bfc-900b-9ec2e8ee9417 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:04.965485 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-159c7308-ef44-4bfc-900b-9ec2e8ee9417 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:04.965485 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:04.965485 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:04.965485 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:04.965485 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:04.965485 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:04.965485 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:04.965485 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:04.965485 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:04.965485 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:04.965485 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:04.965485 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:04.965485 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:04.965485 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:04.965485 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:04.965485 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:04.966397 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:04.966397 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:04.966397 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:04.966397 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:04.966397 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:04.966397 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:04.966397 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:04.966397 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:04.966397 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:04.966397 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:04.966397 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:04.966397 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:04.966397 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:04.966397 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:04.966397 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:04.966397 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:04.966397 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:04.967243 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:04.967243 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:04.967243 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:04.967243 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 243/498] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:04.967460 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-159c7308-ef44-4bfc-900b-9ec2e8ee9417 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:04.979453 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68812b37-1e9e-452c-a385-1db8510369d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:04.980094 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68812b37-1e9e-452c-a385-1db8510369d2 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:04.980325 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68812b37-1e9e-452c-a385-1db8510369d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:04.980591 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68812b37-1e9e-452c-a385-1db8510369d2 None admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:04.981331 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68812b37-1e9e-452c-a385-1db8510369d2 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:04.981633 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68812b37-1e9e-452c-a385-1db8510369d2 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:04.983081 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68812b37-1e9e-452c-a385-1db8510369d2 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:05.021795 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 256/499] 10.222.0.52 () {56 vars in 1133 bytes} [Sun Sep 25 00:42:04 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:05.022769 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0476f1b-f2d6-49f0-9410-5149dcd7687b None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:05.023051 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0476f1b-f2d6-49f0-9410-5149dcd7687b None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:05.023316 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0476f1b-f2d6-49f0-9410-5149dcd7687b None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:05.028758 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 257/500] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:42:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:42:05.028758 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0476f1b-f2d6-49f0-9410-5149dcd7687b None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:05.046407 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:05.051675 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:05.085145 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-159c7308-ef44-4bfc-900b-9ec2e8ee9417 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:05.125489 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ab3c59-bd87-4dec-95ac-47abb50b3c5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:05.126502 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ab3c59-bd87-4dec-95ac-47abb50b3c5f None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:05.126989 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ab3c59-bd87-4dec-95ac-47abb50b3c5f None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:05.127477 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ab3c59-bd87-4dec-95ac-47abb50b3c5f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:05.142098 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0476f1b-f2d6-49f0-9410-5149dcd7687b None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:05.142625 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ab3c59-bd87-4dec-95ac-47abb50b3c5f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:05.144073 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ab3c59-bd87-4dec-95ac-47abb50b3c5f None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:05.158289 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-08ab3c59-bd87-4dec-95ac-47abb50b3c5f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:05.158289 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:05.158289 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:05.158289 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:05.158289 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:05.158289 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:05.158289 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:05.158289 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:05.158289 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:05.158289 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:05.158289 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:05.158289 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:05.158289 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:05.158289 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:05.158289 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:05.158289 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:05.159230 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:05.159230 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:05.159230 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:05.159230 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:05.159230 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:05.159230 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:05.159230 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:05.159230 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:05.159230 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:05.159230 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:05.159230 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:05.159230 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:05.159230 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:05.159230 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:05.159230 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:05.159230 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:05.159230 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:05.160266 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:05.160266 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:05.160266 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:05.160266 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 244/501] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:05.160266 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ab3c59-bd87-4dec-95ac-47abb50b3c5f None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:05.179211 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6b0887-4681-4af0-92f0-f4a70caec301 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:05.180507 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6b0887-4681-4af0-92f0-f4a70caec301 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:05.180759 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6b0887-4681-4af0-92f0-f4a70caec301 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:05.181099 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6b0887-4681-4af0-92f0-f4a70caec301 None admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:05.181777 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad6b0887-4681-4af0-92f0-f4a70caec301 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:05.182075 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad6b0887-4681-4af0-92f0-f4a70caec301 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:05.183449 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad6b0887-4681-4af0-92f0-f4a70caec301 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:05.194427 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 258/502] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:05.195666 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ce83ba-2c83-4d39-911d-ab9a43275e42 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:05.195983 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ce83ba-2c83-4d39-911d-ab9a43275e42 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:05.196245 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ce83ba-2c83-4d39-911d-ab9a43275e42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:05.215467 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-35ce83ba-2c83-4d39-911d-ab9a43275e42 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:42:05.216647 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-35ce83ba-2c83-4d39-911d-ab9a43275e42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:42:05.264458 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ab3c59-bd87-4dec-95ac-47abb50b3c5f None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:05.281710 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:05.283124 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 259/503] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:42:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:42:05.283949 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ce83ba-2c83-4d39-911d-ab9a43275e42 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:05.296773 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:05.299726 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f869d320-7018-4398-9103-aae96229d471 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:05.300694 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f869d320-7018-4398-9103-aae96229d471 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:05.300801 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f869d320-7018-4398-9103-aae96229d471 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:05.301005 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f869d320-7018-4398-9103-aae96229d471 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:05.375863 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f869d320-7018-4398-9103-aae96229d471 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:05.377762 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f869d320-7018-4398-9103-aae96229d471 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:05.397351 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ce83ba-2c83-4d39-911d-ab9a43275e42 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:05.433251 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb32234-fd78-4dfe-beaf-30c0dc716575 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:05.433889 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb32234-fd78-4dfe-beaf-30c0dc716575 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:05.434162 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb32234-fd78-4dfe-beaf-30c0dc716575 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:05.434426 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb32234-fd78-4dfe-beaf-30c0dc716575 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:05.444334 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fb32234-fd78-4dfe-beaf-30c0dc716575 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:42:05.449203 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-3fb32234-fd78-4dfe-beaf-30c0dc716575 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 25 00:42:05.449203 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:05.449203 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:05.449203 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:05.449203 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:05.449203 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:05.449203 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:05.449203 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:05.449203 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:05.449203 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:05.449203 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:05.449203 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:05.449203 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:42:05.449203 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:42:05.449203 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:42:05.449203 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:42:05.450154 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:42:05.450154 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:42:05.450154 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:42:05.450154 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:42:05.450154 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:05.450154 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:05.450154 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:05.450154 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:05.450154 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:05.450154 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:05.450154 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:05.450154 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:05.450154 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:05.450154 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:05.450154 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:05.450154 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:05.450955 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:05.450955 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:05.450955 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:05.450955 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:05.450955 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:42:05.450955 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:42:05.450955 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:42:05.450955 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:42:05.450955 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:42:05.450955 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:42:05.450955 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 25 00:42:05.450955 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:05.450955 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 260/504] 10.222.0.52 () {56 vars in 1102 bytes} [Sun Sep 25 00:42:05 2022] GET /identity/v3/roles/service => generated 85 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:05.451737 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb32234-fd78-4dfe-beaf-30c0dc716575 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:05.497891 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 245/505] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:42:05 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 336 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 00:42:05.497891 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f869d320-7018-4398-9103-aae96229d471 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:05.519003 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:05.537375 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:05.571036 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb32234-fd78-4dfe-beaf-30c0dc716575 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:05.606363 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91916781-68b9-4281-bb9a-231d10c466ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:05.606967 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91916781-68b9-4281-bb9a-231d10c466ba None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:05.607247 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91916781-68b9-4281-bb9a-231d10c466ba None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:05.607512 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91916781-68b9-4281-bb9a-231d10c466ba None admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:05.608142 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91916781-68b9-4281-bb9a-231d10c466ba None admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:05.608438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91916781-68b9-4281-bb9a-231d10c466ba None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:05.609221 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f869d320-7018-4398-9103-aae96229d471 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:05.609844 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91916781-68b9-4281-bb9a-231d10c466ba None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:05.621601 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 261/506] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:05.622404 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91916781-68b9-4281-bb9a-231d10c466ba None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:05.643589 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16dca2b0-1ecc-4518-9706-6a8e2b4d8c7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:05.644215 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16dca2b0-1ecc-4518-9706-6a8e2b4d8c7b None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:05.644490 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16dca2b0-1ecc-4518-9706-6a8e2b4d8c7b None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:05.644755 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16dca2b0-1ecc-4518-9706-6a8e2b4d8c7b None admin] PATH_INFO: `/v3/users/glance` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:05.652691 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16dca2b0-1ecc-4518-9706-6a8e2b4d8c7b None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:05.654145 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16dca2b0-1ecc-4518-9706-6a8e2b4d8c7b None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:05.667072 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-16dca2b0-1ecc-4518-9706-6a8e2b4d8c7b None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 25 00:42:05.667072 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:05.667072 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:05.667072 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:05.667072 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:05.667072 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:05.667072 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:05.667072 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:05.667072 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:05.667072 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:05.667072 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:05.667072 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:05.667072 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 00:42:05.667072 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 00:42:05.667072 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 00:42:05.667072 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 00:42:05.668008 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:05.668008 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:05.668008 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 00:42:05.668008 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:05.668008 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 00:42:05.668008 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:05.668008 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:05.668008 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:05.668008 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:05.668008 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:05.668008 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:05.668008 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:05.668008 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:05.668008 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:05.668008 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:05.668008 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:05.668008 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:05.668008 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:05.669294 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:05.669294 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:05.669294 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 00:42:05.669294 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 00:42:05.669294 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 00:42:05.669294 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 00:42:05.669294 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 00:42:05.669294 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 00:42:05.669294 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 25 00:42:05.669294 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:05.669294 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 246/507] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:42:05 2022] GET /identity/v3/users/glance => generated 84 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:05.669294 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16dca2b0-1ecc-4518-9706-6a8e2b4d8c7b None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:05.725455 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91916781-68b9-4281-bb9a-231d10c466ba None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:05.751423 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:05.754667 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:05.761415 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2dd701c-8db6-4381-837d-faade1974124 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:05.762017 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2dd701c-8db6-4381-837d-faade1974124 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:05.762291 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2dd701c-8db6-4381-837d-faade1974124 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:05.762558 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2dd701c-8db6-4381-837d-faade1974124 None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:05.763207 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2dd701c-8db6-4381-837d-faade1974124 None admin] RBAC: Adding query filter params (name=service) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:05.763510 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2dd701c-8db6-4381-837d-faade1974124 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:05.764950 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2dd701c-8db6-4381-837d-faade1974124 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:05.772741 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 262/508] 10.222.0.52 () {56 vars in 1103 bytes} [Sun Sep 25 00:42:05 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:05.774257 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2dd701c-8db6-4381-837d-faade1974124 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:05.775425 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16dca2b0-1ecc-4518-9706-6a8e2b4d8c7b None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:05.811189 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c073c47-0fca-40a7-964d-a63b796c4087 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:05.811725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c073c47-0fca-40a7-964d-a63b796c4087 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:05.811997 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c073c47-0fca-40a7-964d-a63b796c4087 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:05.812257 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c073c47-0fca-40a7-964d-a63b796c4087 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:05.823770 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c073c47-0fca-40a7-964d-a63b796c4087 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:05.825497 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c073c47-0fca-40a7-964d-a63b796c4087 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:05.839657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-5c073c47-0fca-40a7-964d-a63b796c4087 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:05.839657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:05.839657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:05.839657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:05.839657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:05.839657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:05.839657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:05.839657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:05.839657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:05.839657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:05.839657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:05.839657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:05.839657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:05.839657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:05.839657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:05.839657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:05.840742 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:05.840742 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:05.840742 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:05.840742 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:05.840742 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:05.840742 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:05.840742 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:05.840742 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:05.840742 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:05.840742 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:05.840742 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:05.840742 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:05.840742 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:05.840742 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:05.840742 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:05.840742 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:05.840742 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:05.842039 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:05.842039 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:05.842039 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:05.842039 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 247/509] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:05.842039 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c073c47-0fca-40a7-964d-a63b796c4087 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:05.876336 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2dd701c-8db6-4381-837d-faade1974124 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:05.912233 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a102535-b9e0-4705-b356-e4c1e68dac26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:05.912953 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a102535-b9e0-4705-b356-e4c1e68dac26 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:05.913467 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a102535-b9e0-4705-b356-e4c1e68dac26 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:05.913782 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a102535-b9e0-4705-b356-e4c1e68dac26 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:05.926356 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a102535-b9e0-4705-b356-e4c1e68dac26 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:05.927787 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a102535-b9e0-4705-b356-e4c1e68dac26 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:05.942543 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-8a102535-b9e0-4705-b356-e4c1e68dac26 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:05.942543 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:05.942543 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:05.942543 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:05.942543 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:05.942543 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:05.942543 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:05.942543 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:05.942543 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:05.942543 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:05.942543 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:05.942543 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:05.942543 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:05.942543 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:05.942543 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:05.942543 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:05.943522 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:05.943522 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:05.943522 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:05.943522 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:05.943522 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:05.943522 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:05.943522 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:05.943522 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:05.943522 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:05.943522 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:05.943522 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:05.943522 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:05.943522 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:05.943522 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:05.943522 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:05.943522 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:05.943522 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:05.944421 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:05.944421 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:05.944421 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:05.944421 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 263/510] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:05.944421 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a102535-b9e0-4705-b356-e4c1e68dac26 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:05.952796 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c073c47-0fca-40a7-964d-a63b796c4087 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:05.982384 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:05.988755 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4af1400-539e-4462-b607-50ccb4bc2759 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:05.989405 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4af1400-539e-4462-b607-50ccb4bc2759 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:05.989682 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4af1400-539e-4462-b607-50ccb4bc2759 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:05.989949 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4af1400-539e-4462-b607-50ccb4bc2759 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:05.990689 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:05.999797 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4af1400-539e-4462-b607-50ccb4bc2759 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:06.001226 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4af1400-539e-4462-b607-50ccb4bc2759 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:06.014071 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-e4af1400-539e-4462-b607-50ccb4bc2759 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 25 00:42:06.014071 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:06.014071 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:06.014071 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:06.014071 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:06.014071 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:06.014071 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:06.014071 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:06.014071 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:06.014071 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:06.014071 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:06.014071 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:06.014071 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 00:42:06.014071 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 00:42:06.014071 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 00:42:06.014071 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 00:42:06.014985 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:06.014985 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:06.014985 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 00:42:06.014985 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:06.014985 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 00:42:06.014985 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:06.014985 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:06.014985 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:06.014985 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:06.014985 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:06.014985 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:06.014985 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:06.014985 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:06.014985 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:06.014985 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:06.014985 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:06.014985 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:06.014985 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:06.015896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:06.015896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:06.015896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 00:42:06.015896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 00:42:06.015896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 00:42:06.015896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 00:42:06.015896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 00:42:06.015896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 00:42:06.015896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 25 00:42:06.015896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:06.015896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 248/511] 10.222.0.52 () {56 vars in 1102 bytes} [Sun Sep 25 00:42:05 2022] GET /identity/v3/users/neutron => generated 85 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:06.015896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4af1400-539e-4462-b607-50ccb4bc2759 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:06.050913 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a102535-b9e0-4705-b356-e4c1e68dac26 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:06.085190 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbaed40c-81f4-49ca-8af2-2a3849d7b8e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:06.085753 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbaed40c-81f4-49ca-8af2-2a3849d7b8e7 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:06.086132 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbaed40c-81f4-49ca-8af2-2a3849d7b8e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:06.086506 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbaed40c-81f4-49ca-8af2-2a3849d7b8e7 None admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:06.087218 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbaed40c-81f4-49ca-8af2-2a3849d7b8e7 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:06.087521 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbaed40c-81f4-49ca-8af2-2a3849d7b8e7 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:06.088951 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbaed40c-81f4-49ca-8af2-2a3849d7b8e7 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:06.105895 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 264/512] 10.222.0.52 () {56 vars in 1137 bytes} [Sun Sep 25 00:42:05 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:06.106711 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbaed40c-81f4-49ca-8af2-2a3849d7b8e7 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:06.130945 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4af1400-539e-4462-b607-50ccb4bc2759 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:06.166103 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652dd769-a57c-4f6e-b88e-07bb333571ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:06.166723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652dd769-a57c-4f6e-b88e-07bb333571ac None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:06.167002 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652dd769-a57c-4f6e-b88e-07bb333571ac None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:06.167274 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652dd769-a57c-4f6e-b88e-07bb333571ac None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:06.177739 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-652dd769-a57c-4f6e-b88e-07bb333571ac None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:06.179165 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-652dd769-a57c-4f6e-b88e-07bb333571ac None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:06.193899 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-652dd769-a57c-4f6e-b88e-07bb333571ac None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:06.193899 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:06.193899 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:06.193899 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:06.193899 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:06.193899 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:06.193899 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:06.193899 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:06.193899 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:06.193899 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:06.193899 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:06.193899 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:06.193899 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:06.193899 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:06.193899 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:06.193899 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:06.200553 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:06.200553 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:06.200553 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:06.200553 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:06.200553 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:06.200553 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:06.200553 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:06.200553 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:06.200553 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:06.200553 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:06.200553 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:06.200553 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:06.200553 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:06.200553 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:06.200553 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:06.200553 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:06.200553 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:06.201713 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:06.201713 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:06.201713 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:06.201713 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 249/513] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:06.201713 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652dd769-a57c-4f6e-b88e-07bb333571ac None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:06.221182 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbaed40c-81f4-49ca-8af2-2a3849d7b8e7 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:06.222848 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:06.240100 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:06.256901 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed9a8753-8a69-42e9-ac96-2a7aedb06cba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:06.257526 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9a8753-8a69-42e9-ac96-2a7aedb06cba None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:06.257871 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9a8753-8a69-42e9-ac96-2a7aedb06cba None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:06.258146 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9a8753-8a69-42e9-ac96-2a7aedb06cba None admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:06.258772 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed9a8753-8a69-42e9-ac96-2a7aedb06cba None admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:06.259067 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed9a8753-8a69-42e9-ac96-2a7aedb06cba None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:06.260438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed9a8753-8a69-42e9-ac96-2a7aedb06cba None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:06.271318 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 265/514] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:06.272185 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed9a8753-8a69-42e9-ac96-2a7aedb06cba None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:06.299135 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652dd769-a57c-4f6e-b88e-07bb333571ac None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:06.335021 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee249d70-eea5-4791-a9f0-c9f580fef101 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:06.335680 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee249d70-eea5-4791-a9f0-c9f580fef101 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:06.336464 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee249d70-eea5-4791-a9f0-c9f580fef101 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:06.336739 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee249d70-eea5-4791-a9f0-c9f580fef101 None admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:06.337600 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee249d70-eea5-4791-a9f0-c9f580fef101 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:06.337889 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee249d70-eea5-4791-a9f0-c9f580fef101 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:06.339246 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee249d70-eea5-4791-a9f0-c9f580fef101 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:06.351224 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 250/515] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:06.352628 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee249d70-eea5-4791-a9f0-c9f580fef101 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:06.375876 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed9a8753-8a69-42e9-ac96-2a7aedb06cba None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:06.410601 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc879ffd-0255-4e3d-9003-c45abda64387 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:06.411201 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc879ffd-0255-4e3d-9003-c45abda64387 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:06.411472 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc879ffd-0255-4e3d-9003-c45abda64387 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:06.411736 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc879ffd-0255-4e3d-9003-c45abda64387 None admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:06.412452 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc879ffd-0255-4e3d-9003-c45abda64387 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:06.412752 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc879ffd-0255-4e3d-9003-c45abda64387 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:06.414851 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc879ffd-0255-4e3d-9003-c45abda64387 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:06.432550 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 266/516] 10.222.0.52 () {56 vars in 1139 bytes} [Sun Sep 25 00:42:06 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:06.433376 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc879ffd-0255-4e3d-9003-c45abda64387 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:06.455403 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:06.462939 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee249d70-eea5-4791-a9f0-c9f580fef101 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:06.464600 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:06.500855 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c39662b6-4cfd-4531-b282-e529acf87fd1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:06.503428 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39662b6-4cfd-4531-b282-e529acf87fd1 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:06.503785 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39662b6-4cfd-4531-b282-e529acf87fd1 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:06.504616 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39662b6-4cfd-4531-b282-e529acf87fd1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:06.540500 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc879ffd-0255-4e3d-9003-c45abda64387 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:06.574135 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78ce57e5-5b35-47f9-935d-2e93dcd4369d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:06.574794 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ce57e5-5b35-47f9-935d-2e93dcd4369d None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:06.575061 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ce57e5-5b35-47f9-935d-2e93dcd4369d None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:06.575321 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ce57e5-5b35-47f9-935d-2e93dcd4369d None admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:06.575943 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78ce57e5-5b35-47f9-935d-2e93dcd4369d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:06.576234 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78ce57e5-5b35-47f9-935d-2e93dcd4369d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:06.577609 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78ce57e5-5b35-47f9-935d-2e93dcd4369d None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:06.582925 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c39662b6-4cfd-4531-b282-e529acf87fd1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:06.584372 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c39662b6-4cfd-4531-b282-e529acf87fd1 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:06.590024 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 267/517] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:06.590777 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78ce57e5-5b35-47f9-935d-2e93dcd4369d None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:06.669763 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:06.683432 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 251/518] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:42:06 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 332 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 00:42:06.684249 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c39662b6-4cfd-4531-b282-e529acf87fd1 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:06.686706 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:06.693415 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78ce57e5-5b35-47f9-935d-2e93dcd4369d None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:06.730307 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f065c1c-6f3b-471a-a394-611cbdea3b92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:06.730937 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f065c1c-6f3b-471a-a394-611cbdea3b92 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:06.731205 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f065c1c-6f3b-471a-a394-611cbdea3b92 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:06.731470 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f065c1c-6f3b-471a-a394-611cbdea3b92 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:06.738918 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f065c1c-6f3b-471a-a394-611cbdea3b92 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:06.740355 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f065c1c-6f3b-471a-a394-611cbdea3b92 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:06.752720 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-0f065c1c-6f3b-471a-a394-611cbdea3b92 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:42:06.752720 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:06.752720 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:06.752720 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:06.752720 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:06.752720 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:06.752720 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:06.752720 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:06.752720 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:06.752720 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:06.752720 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:06.752720 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:06.752720 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 00:42:06.752720 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 00:42:06.752720 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 00:42:06.753625 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 00:42:06.753625 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:06.753625 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:06.753625 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:06.753625 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:06.753625 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:06.753625 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:06.753625 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:06.753625 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:06.753625 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:06.753625 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:06.753625 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:06.753625 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:06.753625 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:06.753625 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:06.753625 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:06.753625 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:06.754458 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 00:42:06.754458 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 00:42:06.754458 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 00:42:06.754458 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 00:42:06.754458 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 00:42:06.754458 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 00:42:06.754458 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:42:06.754458 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:06.754458 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 268/519] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:42:06 2022] GET /identity/v3/projects/service => generated 88 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:06.754946 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f065c1c-6f3b-471a-a394-611cbdea3b92 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:06.793548 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c39662b6-4cfd-4531-b282-e529acf87fd1 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:06.826924 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c39a9211-9ef6-47f2-a241-d09bbb6f6780 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:06.827529 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39a9211-9ef6-47f2-a241-d09bbb6f6780 None admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:06.827803 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39a9211-9ef6-47f2-a241-d09bbb6f6780 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:06.828068 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39a9211-9ef6-47f2-a241-d09bbb6f6780 None admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:06.828708 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c39a9211-9ef6-47f2-a241-d09bbb6f6780 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:06.830150 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c39a9211-9ef6-47f2-a241-d09bbb6f6780 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:06.850576 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 252/520] 10.222.0.52 () {60 vars in 1132 bytes} [Sun Sep 25 00:42:06 2022] POST /identity/v3/users => generated 247 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:42:06.851340 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c39a9211-9ef6-47f2-a241-d09bbb6f6780 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:06.853638 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f065c1c-6f3b-471a-a394-611cbdea3b92 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:06.895009 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32386dcc-ee58-4bcc-a0d4-d7457717ed1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:06.895789 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32386dcc-ee58-4bcc-a0d4-d7457717ed1f None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:06.896149 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32386dcc-ee58-4bcc-a0d4-d7457717ed1f None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:06.896502 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32386dcc-ee58-4bcc-a0d4-d7457717ed1f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:06.913070 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:06.933092 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:06.984219 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32386dcc-ee58-4bcc-a0d4-d7457717ed1f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:06.984817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c39a9211-9ef6-47f2-a241-d09bbb6f6780 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:06.985908 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32386dcc-ee58-4bcc-a0d4-d7457717ed1f None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:07.028094 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b23f8142-f328-407c-847d-b10c1724fed4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:07.028565 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b23f8142-f328-407c-847d-b10c1724fed4 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:07.028845 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b23f8142-f328-407c-847d-b10c1724fed4 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:07.029112 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b23f8142-f328-407c-847d-b10c1724fed4 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:07.038615 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b23f8142-f328-407c-847d-b10c1724fed4 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:07.039646 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b23f8142-f328-407c-847d-b10c1724fed4 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:07.051809 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-b23f8142-f328-407c-847d-b10c1724fed4 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 25 00:42:07.051809 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:07.051809 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:07.051809 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:07.051809 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:07.051809 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:07.051809 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:07.051809 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:07.051809 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:07.051809 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:07.051809 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:07.051809 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:07.051809 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 00:42:07.051809 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 00:42:07.051809 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 00:42:07.051809 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 00:42:07.052845 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:07.052845 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:07.052845 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 00:42:07.052845 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:07.052845 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 00:42:07.052845 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:07.052845 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:07.052845 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:07.052845 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:07.052845 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:07.052845 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:07.052845 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:07.052845 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:07.052845 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:07.052845 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:07.052845 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:07.052845 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:07.052845 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:07.053820 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:07.053820 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:07.053820 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 00:42:07.053820 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 00:42:07.053820 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 00:42:07.053820 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 00:42:07.053820 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 00:42:07.053820 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 00:42:07.053820 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 25 00:42:07.053820 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:07.053820 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 253/521] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:42:06 2022] GET /identity/v3/users/cinder => generated 84 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:07.054389 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b23f8142-f328-407c-847d-b10c1724fed4 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:07.113506 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 269/522] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:42:06 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 359 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 00:42:07.114395 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32386dcc-ee58-4bcc-a0d4-d7457717ed1f None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:07.160877 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:07.174257 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b23f8142-f328-407c-847d-b10c1724fed4 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:07.195112 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:07.214405 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cb32932-20ed-4324-ac69-83914636ffb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:07.215032 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb32932-20ed-4324-ac69-83914636ffb6 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:07.215311 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb32932-20ed-4324-ac69-83914636ffb6 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:07.216007 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb32932-20ed-4324-ac69-83914636ffb6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:07.227486 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cb32932-20ed-4324-ac69-83914636ffb6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:07.228791 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32386dcc-ee58-4bcc-a0d4-d7457717ed1f None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:07.228917 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cb32932-20ed-4324-ac69-83914636ffb6 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:07.244927 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-9cb32932-20ed-4324-ac69-83914636ffb6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:07.244927 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:07.244927 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:07.244927 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:07.244927 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:07.244927 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:07.244927 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:07.244927 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:07.244927 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:07.244927 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:07.244927 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:07.244927 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:07.244927 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:07.244927 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:07.244927 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:07.244927 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:07.245947 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:07.245947 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:07.245947 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:07.245947 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:07.245947 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:07.245947 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:07.245947 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:07.245947 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:07.245947 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:07.245947 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:07.245947 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:07.245947 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:07.245947 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:07.245947 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:07.245947 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:07.245947 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:07.245947 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:07.246819 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:07.246819 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:07.246819 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:07.246819 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 254/523] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:07.246819 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cb32932-20ed-4324-ac69-83914636ffb6 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:07.273224 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2812c57-b3af-41f1-bf7b-26c3defb9b8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:07.273946 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2812c57-b3af-41f1-bf7b-26c3defb9b8c None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:07.274222 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2812c57-b3af-41f1-bf7b-26c3defb9b8c None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:07.274793 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2812c57-b3af-41f1-bf7b-26c3defb9b8c None admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:07.275173 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2812c57-b3af-41f1-bf7b-26c3defb9b8c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:07.275473 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2812c57-b3af-41f1-bf7b-26c3defb9b8c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:07.276915 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2812c57-b3af-41f1-bf7b-26c3defb9b8c None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:07.296121 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 270/524] 10.222.0.52 () {56 vars in 1148 bytes} [Sun Sep 25 00:42:07 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:07.301738 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2812c57-b3af-41f1-bf7b-26c3defb9b8c None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:07.358495 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cb32932-20ed-4324-ac69-83914636ffb6 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:07.393250 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-674aeffc-8b55-4952-ad51-69a196f85304 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:07.393804 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674aeffc-8b55-4952-ad51-69a196f85304 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:07.394080 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674aeffc-8b55-4952-ad51-69a196f85304 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:07.394355 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674aeffc-8b55-4952-ad51-69a196f85304 None admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:07.395079 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-674aeffc-8b55-4952-ad51-69a196f85304 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:07.395381 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-674aeffc-8b55-4952-ad51-69a196f85304 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:07.396801 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-674aeffc-8b55-4952-ad51-69a196f85304 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:07.410886 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:07.414827 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2812c57-b3af-41f1-bf7b-26c3defb9b8c None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:07.438623 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 255/525] 10.222.0.52 () {56 vars in 1137 bytes} [Sun Sep 25 00:42:07 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:07.438623 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-674aeffc-8b55-4952-ad51-69a196f85304 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:07.455112 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f21933d-7b6c-4e7c-ad47-190c35b9093d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:07.455794 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f21933d-7b6c-4e7c-ad47-190c35b9093d None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:07.456014 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f21933d-7b6c-4e7c-ad47-190c35b9093d None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:07.456285 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f21933d-7b6c-4e7c-ad47-190c35b9093d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:07.467950 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f21933d-7b6c-4e7c-ad47-190c35b9093d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:07.468728 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:07.470549 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f21933d-7b6c-4e7c-ad47-190c35b9093d None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:07.493677 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-7f21933d-7b6c-4e7c-ad47-190c35b9093d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:07.493677 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:07.493677 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:07.493677 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:07.493677 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:07.493677 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:07.493677 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:07.493677 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:07.493677 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:07.493677 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:07.493677 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:07.493677 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:07.493677 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:07.493677 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:07.493677 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:07.493677 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:07.494629 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:07.494629 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:07.494629 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:07.494629 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:07.494629 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:07.494629 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:07.494629 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:07.494629 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:07.494629 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:07.494629 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:07.494629 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:07.494629 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:07.494629 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:07.494629 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:07.494629 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:07.494629 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:07.494629 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:07.495525 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:07.495525 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:07.495525 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:07.495696 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 271/526] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:07.496667 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f21933d-7b6c-4e7c-ad47-190c35b9093d None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:07.549009 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-674aeffc-8b55-4952-ad51-69a196f85304 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:07.588275 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-484441c1-8237-4b88-a522-c8faadb7edc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:07.588882 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484441c1-8237-4b88-a522-c8faadb7edc2 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:07.589161 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484441c1-8237-4b88-a522-c8faadb7edc2 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:07.589797 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484441c1-8237-4b88-a522-c8faadb7edc2 None admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:07.590104 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-484441c1-8237-4b88-a522-c8faadb7edc2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:07.590408 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-484441c1-8237-4b88-a522-c8faadb7edc2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:07.591851 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-484441c1-8237-4b88-a522-c8faadb7edc2 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:07.603780 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 256/527] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:07.604731 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-484441c1-8237-4b88-a522-c8faadb7edc2 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:07.622201 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f21933d-7b6c-4e7c-ad47-190c35b9093d None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:07.663315 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8183eb0f-9e65-4558-8826-e122cf0e7094 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:07.663838 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8183eb0f-9e65-4558-8826-e122cf0e7094 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:07.664683 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8183eb0f-9e65-4558-8826-e122cf0e7094 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:07.665296 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8183eb0f-9e65-4558-8826-e122cf0e7094 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:07.665856 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8183eb0f-9e65-4558-8826-e122cf0e7094 None admin] RBAC: Adding query filter params (role.id=a8e34e460e8a4ed7bcaa54da3c2d5a0c, scope.project.id=9db0fd07fc494203b2ada13f3cd046d3, user.id=f142988a4dd14a5f96eacea45c046f85) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:07.666160 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8183eb0f-9e65-4558-8826-e122cf0e7094 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:07.667595 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8183eb0f-9e65-4558-8826-e122cf0e7094 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:07.681301 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 272/528] 10.222.0.52 () {56 vars in 1374 bytes} [Sun Sep 25 00:42:07 2022] GET /identity/v3/role_assignments?user.id=f142988a4dd14a5f96eacea45c046f85&scope.project.id=9db0fd07fc494203b2ada13f3cd046d3&role.id=a8e34e460e8a4ed7bcaa54da3c2d5a0c => generated 259 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:07.681301 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8183eb0f-9e65-4558-8826-e122cf0e7094 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:07.700625 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:07.722705 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:07.738915 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-484441c1-8237-4b88-a522-c8faadb7edc2 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:07.779863 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e2fe7c-cf44-4c02-a384-033db7fd6b86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:07.780973 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e2fe7c-cf44-4c02-a384-033db7fd6b86 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:07.781092 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e2fe7c-cf44-4c02-a384-033db7fd6b86 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:07.781510 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e2fe7c-cf44-4c02-a384-033db7fd6b86 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:07.814501 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8183eb0f-9e65-4558-8826-e122cf0e7094 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:07.856841 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51072a3-4f25-4b97-9300-e9e8e9d37385 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:07.857710 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51072a3-4f25-4b97-9300-e9e8e9d37385 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:07.858032 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51072a3-4f25-4b97-9300-e9e8e9d37385 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:07.858404 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51072a3-4f25-4b97-9300-e9e8e9d37385 None admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:07.859159 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e51072a3-4f25-4b97-9300-e9e8e9d37385 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:07.859570 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e51072a3-4f25-4b97-9300-e9e8e9d37385 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:07.861147 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e51072a3-4f25-4b97-9300-e9e8e9d37385 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:07.878060 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00e2fe7c-cf44-4c02-a384-033db7fd6b86 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:07.879509 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00e2fe7c-cf44-4c02-a384-033db7fd6b86 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:07.880265 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 273/529] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:07.881149 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51072a3-4f25-4b97-9300-e9e8e9d37385 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:07.973369 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:08.002503 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51072a3-4f25-4b97-9300-e9e8e9d37385 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:08.008661 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:08.027237 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 257/530] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:42:07 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 418 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 00:42:08.027237 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e2fe7c-cf44-4c02-a384-033db7fd6b86 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:08.052959 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e28224a7-13ee-4138-a577-82ca4bb1c7cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:08.053864 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28224a7-13ee-4138-a577-82ca4bb1c7cd None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:08.054154 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28224a7-13ee-4138-a577-82ca4bb1c7cd None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:08.054555 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28224a7-13ee-4138-a577-82ca4bb1c7cd None admin] PATH_INFO: `/v3/projects/service` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:08.062336 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e28224a7-13ee-4138-a577-82ca4bb1c7cd None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:08.063806 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e28224a7-13ee-4138-a577-82ca4bb1c7cd None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:08.077719 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-e28224a7-13ee-4138-a577-82ca4bb1c7cd None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:42:08.077719 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:08.077719 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:08.077719 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:08.077719 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:08.077719 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:08.077719 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:08.077719 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:08.077719 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:08.077719 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:08.077719 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:08.077719 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:08.077719 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 00:42:08.077719 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 00:42:08.077719 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 00:42:08.078824 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 00:42:08.078824 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:08.078824 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:08.078824 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:08.078824 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:08.078824 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:08.078824 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:08.078824 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:08.078824 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:08.078824 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:08.078824 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:08.078824 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:08.078824 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:08.078824 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:08.078824 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:08.078824 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:08.078824 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:08.079905 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 00:42:08.079905 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 00:42:08.079905 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 00:42:08.079905 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 00:42:08.079905 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 00:42:08.079905 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 00:42:08.079905 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:42:08.079905 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:08.081540 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 274/531] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:42:07 2022] GET /identity/v3/projects/service => generated 88 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:08.082157 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e28224a7-13ee-4138-a577-82ca4bb1c7cd None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:08.157291 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e2fe7c-cf44-4c02-a384-033db7fd6b86 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:08.203330 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e28224a7-13ee-4138-a577-82ca4bb1c7cd None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:08.207451 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-118e33d6-7eb8-40bc-8e94-7b3f1e19f6b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:08.208327 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-118e33d6-7eb8-40bc-8e94-7b3f1e19f6b2 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:08.208798 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-118e33d6-7eb8-40bc-8e94-7b3f1e19f6b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:08.209208 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-118e33d6-7eb8-40bc-8e94-7b3f1e19f6b2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:08.217640 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-118e33d6-7eb8-40bc-8e94-7b3f1e19f6b2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:08.219298 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-118e33d6-7eb8-40bc-8e94-7b3f1e19f6b2 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:08.232919 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-118e33d6-7eb8-40bc-8e94-7b3f1e19f6b2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:42:08.232919 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:08.232919 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:08.232919 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:08.232919 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:08.232919 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:08.232919 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:08.232919 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:08.232919 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:08.232919 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:08.232919 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:08.232919 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:08.232919 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 00:42:08.232919 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 00:42:08.232919 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 00:42:08.233985 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 00:42:08.233985 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:08.233985 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:08.233985 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:08.233985 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:08.233985 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:08.233985 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:08.233985 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:08.233985 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:08.233985 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:08.233985 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:08.233985 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:08.233985 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:08.233985 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:08.233985 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:08.233985 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:08.233985 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:08.234989 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 00:42:08.234989 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 00:42:08.234989 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 00:42:08.234989 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 00:42:08.234989 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 00:42:08.234989 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 00:42:08.234989 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:42:08.234989 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:08.237139 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 258/532] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:42:08 2022] GET /identity/v3/projects/service => generated 88 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:08.238560 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-118e33d6-7eb8-40bc-8e94-7b3f1e19f6b2 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:08.240279 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4658b5db-8555-4c86-a409-c4abb12d10b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:08.240972 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4658b5db-8555-4c86-a409-c4abb12d10b2 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:08.241323 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4658b5db-8555-4c86-a409-c4abb12d10b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:08.241594 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4658b5db-8555-4c86-a409-c4abb12d10b2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:08.245431 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:08.254360 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4658b5db-8555-4c86-a409-c4abb12d10b2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:08.255951 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4658b5db-8555-4c86-a409-c4abb12d10b2 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:08.271270 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-4658b5db-8555-4c86-a409-c4abb12d10b2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:08.271270 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:08.271270 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:08.271270 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:08.271270 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:08.271270 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:08.271270 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:08.271270 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:08.271270 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:08.271270 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:08.271270 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:08.271270 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:08.271270 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:08.271270 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:08.271270 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:08.271270 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:08.272432 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:08.272432 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:08.272432 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:08.272432 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:08.272432 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:08.272432 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:08.272432 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:08.272432 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:08.272432 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:08.272432 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:08.272432 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:08.272432 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:08.272432 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:08.272432 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:08.272432 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:08.272432 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:08.272432 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:08.274138 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:08.274138 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:08.274138 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:08.275197 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 275/533] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:08.276066 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4658b5db-8555-4c86-a409-c4abb12d10b2 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:08.306836 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:08.374281 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-118e33d6-7eb8-40bc-8e94-7b3f1e19f6b2 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:08.404464 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4658b5db-8555-4c86-a409-c4abb12d10b2 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:08.421036 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cf7cf5e-57c2-4066-a3e2-ae5d3a74d369 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:08.421975 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf7cf5e-57c2-4066-a3e2-ae5d3a74d369 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:08.422549 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf7cf5e-57c2-4066-a3e2-ae5d3a74d369 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:08.423015 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf7cf5e-57c2-4066-a3e2-ae5d3a74d369 None admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:08.424131 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cf7cf5e-57c2-4066-a3e2-ae5d3a74d369 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:08.424608 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cf7cf5e-57c2-4066-a3e2-ae5d3a74d369 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:08.426653 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cf7cf5e-57c2-4066-a3e2-ae5d3a74d369 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:08.441786 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ac8c9f-fc5c-471b-a5dc-ec994ff1a85b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:08.442134 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ac8c9f-fc5c-471b-a5dc-ec994ff1a85b None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:08.442486 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ac8c9f-fc5c-471b-a5dc-ec994ff1a85b None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:08.442822 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ac8c9f-fc5c-471b-a5dc-ec994ff1a85b None admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:08.443565 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ac8c9f-fc5c-471b-a5dc-ec994ff1a85b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:08.443946 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ac8c9f-fc5c-471b-a5dc-ec994ff1a85b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:08.446102 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ac8c9f-fc5c-471b-a5dc-ec994ff1a85b None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:08.451896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 259/534] 10.222.0.52 () {56 vars in 1148 bytes} [Sun Sep 25 00:42:08 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:08.452995 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cf7cf5e-57c2-4066-a3e2-ae5d3a74d369 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:08.460567 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 276/535] 10.222.0.52 () {56 vars in 1148 bytes} [Sun Sep 25 00:42:08 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:08.461950 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ac8c9f-fc5c-471b-a5dc-ec994ff1a85b None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:08.533153 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:08.585401 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cf7cf5e-57c2-4066-a3e2-ae5d3a74d369 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:08.592946 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ac8c9f-fc5c-471b-a5dc-ec994ff1a85b None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:08.598540 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:08.631121 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b83869-e4a2-4109-b846-d15a0706a58e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:08.632133 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b83869-e4a2-4109-b846-d15a0706a58e None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:08.632485 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab536c8-89aa-4da1-b971-f28b07e42e81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:08.632817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b83869-e4a2-4109-b846-d15a0706a58e None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:08.633290 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab536c8-89aa-4da1-b971-f28b07e42e81 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:08.633640 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab536c8-89aa-4da1-b971-f28b07e42e81 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:08.633910 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab536c8-89aa-4da1-b971-f28b07e42e81 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:08.634235 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b83869-e4a2-4109-b846-d15a0706a58e None admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:08.634805 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab536c8-89aa-4da1-b971-f28b07e42e81 None admin] RBAC: Adding query filter params (role.id=a8e34e460e8a4ed7bcaa54da3c2d5a0c, scope.project.id=9db0fd07fc494203b2ada13f3cd046d3, user.id=4ebfc0abf1b842459a44ca9324252688) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:08.635170 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab536c8-89aa-4da1-b971-f28b07e42e81 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:08.635403 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b83869-e4a2-4109-b846-d15a0706a58e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:08.635879 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b83869-e4a2-4109-b846-d15a0706a58e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:08.636805 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab536c8-89aa-4da1-b971-f28b07e42e81 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:08.637635 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b83869-e4a2-4109-b846-d15a0706a58e None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:08.643956 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 277/536] 10.222.0.52 () {56 vars in 1374 bytes} [Sun Sep 25 00:42:08 2022] GET /identity/v3/role_assignments?user.id=4ebfc0abf1b842459a44ca9324252688&scope.project.id=9db0fd07fc494203b2ada13f3cd046d3&role.id=a8e34e460e8a4ed7bcaa54da3c2d5a0c => generated 259 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:08.645000 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab536c8-89aa-4da1-b971-f28b07e42e81 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:08.655177 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 260/537] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:08.660535 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b83869-e4a2-4109-b846-d15a0706a58e None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:08.789239 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab536c8-89aa-4da1-b971-f28b07e42e81 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:08.804879 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:08.831631 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-440df593-50b7-4cbc-8493-6b6a3d8de3dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:08.832622 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440df593-50b7-4cbc-8493-6b6a3d8de3dc None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:08.833164 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440df593-50b7-4cbc-8493-6b6a3d8de3dc None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:08.833662 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440df593-50b7-4cbc-8493-6b6a3d8de3dc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:08.834576 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-440df593-50b7-4cbc-8493-6b6a3d8de3dc None admin] RBAC: Adding query filter params (role.id=a8e34e460e8a4ed7bcaa54da3c2d5a0c, scope.project.id=9db0fd07fc494203b2ada13f3cd046d3, user.id=20c274e96dec478b98ef51f80ea893b0) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:08.835106 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-440df593-50b7-4cbc-8493-6b6a3d8de3dc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:08.837535 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-440df593-50b7-4cbc-8493-6b6a3d8de3dc None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:08.844915 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 278/538] 10.222.0.52 () {56 vars in 1374 bytes} [Sun Sep 25 00:42:08 2022] GET /identity/v3/role_assignments?user.id=20c274e96dec478b98ef51f80ea893b0&scope.project.id=9db0fd07fc494203b2ada13f3cd046d3&role.id=a8e34e460e8a4ed7bcaa54da3c2d5a0c => generated 259 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:08.900725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b83869-e4a2-4109-b846-d15a0706a58e None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:08.957645 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d16309c6-5c17-40f2-8112-f66b6a81f5b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:08.958355 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16309c6-5c17-40f2-8112-f66b6a81f5b6 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:08.958627 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16309c6-5c17-40f2-8112-f66b6a81f5b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:08.958943 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16309c6-5c17-40f2-8112-f66b6a81f5b6 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:08.967403 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d16309c6-5c17-40f2-8112-f66b6a81f5b6 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:08.968945 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d16309c6-5c17-40f2-8112-f66b6a81f5b6 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:08.982977 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-d16309c6-5c17-40f2-8112-f66b6a81f5b6 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:42:08.982977 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:08.982977 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:08.982977 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:08.982977 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:08.982977 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:08.982977 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:08.982977 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:08.982977 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:08.982977 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:08.982977 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:08.982977 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:08.982977 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 00:42:08.982977 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 00:42:08.982977 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 00:42:08.983896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 00:42:08.983896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:08.983896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:08.983896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:08.983896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:08.983896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:08.983896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:08.983896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:08.983896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:08.983896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:08.983896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:08.983896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:08.983896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:08.983896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:08.983896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:08.983896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:08.983896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:08.984745 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 00:42:08.984745 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 00:42:08.984745 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 00:42:08.984745 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 00:42:08.984745 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 00:42:08.984745 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 00:42:08.984745 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:42:08.984745 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:08.984745 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 261/539] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:42:08 2022] GET /identity/v3/projects/service => generated 88 bytes in 324 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:08.990294 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-440df593-50b7-4cbc-8493-6b6a3d8de3dc None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:09.139429 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-440df593-50b7-4cbc-8493-6b6a3d8de3dc None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:09.194605 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4767dae1-1cba-4e7b-bca7-64ab7835ab73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:09.196087 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4767dae1-1cba-4e7b-bca7-64ab7835ab73 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:09.196507 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4767dae1-1cba-4e7b-bca7-64ab7835ab73 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:09.196941 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4767dae1-1cba-4e7b-bca7-64ab7835ab73 None admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:09.197809 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4767dae1-1cba-4e7b-bca7-64ab7835ab73 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:09.198250 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4767dae1-1cba-4e7b-bca7-64ab7835ab73 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:09.199848 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4767dae1-1cba-4e7b-bca7-64ab7835ab73 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:09.218565 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 279/540] 10.222.0.52 () {56 vars in 1148 bytes} [Sun Sep 25 00:42:08 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:09.224810 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d16309c6-5c17-40f2-8112-f66b6a81f5b6 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:09.274815 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:09.341640 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b693c5-3d9d-4119-9ed1-2800166bb453 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:09.341909 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b693c5-3d9d-4119-9ed1-2800166bb453 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:09.342175 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b693c5-3d9d-4119-9ed1-2800166bb453 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:09.343613 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 280/541] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:42:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 00:42:09.349932 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c5ce03-af3b-4709-9fab-f11cc8cf5fe0 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:09.350384 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c5ce03-af3b-4709-9fab-f11cc8cf5fe0 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:09.350817 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c5ce03-af3b-4709-9fab-f11cc8cf5fe0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:09.372691 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-27c5ce03-af3b-4709-9fab-f11cc8cf5fe0 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:42:09.374189 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-27c5ce03-af3b-4709-9fab-f11cc8cf5fe0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:42:09.393382 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d16309c6-5c17-40f2-8112-f66b6a81f5b6 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:09.444665 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:09.468533 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d30c098-70e9-4421-9d94-8ed59bb0e328 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:09.469141 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d30c098-70e9-4421-9d94-8ed59bb0e328 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:09.469447 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d30c098-70e9-4421-9d94-8ed59bb0e328 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:09.469715 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d30c098-70e9-4421-9d94-8ed59bb0e328 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:09.470441 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d30c098-70e9-4421-9d94-8ed59bb0e328 None admin] RBAC: Adding query filter params (role.id=a8e34e460e8a4ed7bcaa54da3c2d5a0c, scope.project.id=9db0fd07fc494203b2ada13f3cd046d3, user.id=944f10f38ee2428a85599f6a714ac806) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:09.470747 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d30c098-70e9-4421-9d94-8ed59bb0e328 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:09.472218 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d30c098-70e9-4421-9d94-8ed59bb0e328 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:09.491319 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 262/542] 10.222.0.52 () {56 vars in 1374 bytes} [Sun Sep 25 00:42:09 2022] GET /identity/v3/role_assignments?user.id=944f10f38ee2428a85599f6a714ac806&scope.project.id=9db0fd07fc494203b2ada13f3cd046d3&role.id=a8e34e460e8a4ed7bcaa54da3c2d5a0c => generated 259 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:09.522608 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 281/543] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:42:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 25 00:42:09.955559 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d30c098-70e9-4421-9d94-8ed59bb0e328 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:10.070923 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:10.082638 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d30c098-70e9-4421-9d94-8ed59bb0e328 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:10.123306 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5662f19-5ee5-49a7-a00d-49ea89f57398 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:10.123994 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5662f19-5ee5-49a7-a00d-49ea89f57398 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:10.124272 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5662f19-5ee5-49a7-a00d-49ea89f57398 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:10.124540 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5662f19-5ee5-49a7-a00d-49ea89f57398 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:10.135843 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5662f19-5ee5-49a7-a00d-49ea89f57398 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:42:10.141450 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-a5662f19-5ee5-49a7-a00d-49ea89f57398 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 25 00:42:10.141450 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:10.141450 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:10.141450 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:10.141450 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:10.141450 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:10.141450 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:10.141450 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:10.141450 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:10.141450 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:10.141450 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:10.141450 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:10.141450 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:42:10.141450 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:42:10.141450 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:42:10.141450 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:42:10.142386 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:42:10.142386 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:42:10.142386 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:42:10.142386 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:42:10.142386 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:10.142386 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:10.142386 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:10.142386 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:10.142386 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:10.142386 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:10.142386 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:10.142386 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:10.142386 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:10.142386 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:10.142386 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:10.142386 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:10.143186 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:10.143186 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:10.143186 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:10.143186 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:10.143186 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:42:10.143186 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:42:10.143186 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:42:10.143186 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:42:10.143186 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:42:10.143186 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:42:10.143186 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 25 00:42:10.143186 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:10.143186 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 263/544] 10.222.0.52 () {56 vars in 1102 bytes} [Sun Sep 25 00:42:09 2022] GET /identity/v3/roles/service => generated 85 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:10.149043 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c5ce03-af3b-4709-9fab-f11cc8cf5fe0 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:10.269960 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c5ce03-af3b-4709-9fab-f11cc8cf5fe0 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:10.299181 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e7bf45-edf2-4bb1-9caf-2ef4f74d1b8a None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:10.300155 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e7bf45-edf2-4bb1-9caf-2ef4f74d1b8a None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:10.300644 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e7bf45-edf2-4bb1-9caf-2ef4f74d1b8a None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:10.302202 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 264/545] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:42:10 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 00:42:10.306728 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe6b3f7-0123-48bd-9108-5452bbd9b6be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:10.307332 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe6b3f7-0123-48bd-9108-5452bbd9b6be None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:10.307603 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe6b3f7-0123-48bd-9108-5452bbd9b6be None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:10.307889 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe6b3f7-0123-48bd-9108-5452bbd9b6be None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:10.308402 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704a1123-94e7-4762-8a84-c3188ca6fc88 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:10.308613 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe6b3f7-0123-48bd-9108-5452bbd9b6be None admin] RBAC: Adding query filter params (name=service) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:10.308918 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe6b3f7-0123-48bd-9108-5452bbd9b6be None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:10.309338 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704a1123-94e7-4762-8a84-c3188ca6fc88 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:10.309802 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704a1123-94e7-4762-8a84-c3188ca6fc88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:10.310311 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe6b3f7-0123-48bd-9108-5452bbd9b6be None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:10.319141 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 282/546] 10.222.0.52 () {56 vars in 1103 bytes} [Sun Sep 25 00:42:10 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:42:10.326519 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe6b3f7-0123-48bd-9108-5452bbd9b6be None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:10.330288 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-704a1123-94e7-4762-8a84-c3188ca6fc88 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:42:10.331702 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-704a1123-94e7-4762-8a84-c3188ca6fc88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:42:10.377631 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:10.434348 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 265/547] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:42:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 25 00:42:10.518952 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe6b3f7-0123-48bd-9108-5452bbd9b6be None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:10.564809 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d5ecad-5c11-4804-a625-cb161dbb05a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:10.565509 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d5ecad-5c11-4804-a625-cb161dbb05a6 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:10.565797 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d5ecad-5c11-4804-a625-cb161dbb05a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:10.566069 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d5ecad-5c11-4804-a625-cb161dbb05a6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:10.580949 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68d5ecad-5c11-4804-a625-cb161dbb05a6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:10.582442 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68d5ecad-5c11-4804-a625-cb161dbb05a6 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:10.602669 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-68d5ecad-5c11-4804-a625-cb161dbb05a6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:10.602669 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:10.602669 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:10.602669 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:10.602669 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:10.602669 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:10.602669 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:10.602669 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:10.602669 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:10.602669 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:10.602669 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:10.602669 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:10.602669 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:10.602669 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:10.602669 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:10.602669 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:10.603624 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:10.603624 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:10.603624 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:10.603624 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:10.603624 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:10.603624 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:10.603624 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:10.603624 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:10.603624 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:10.603624 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:10.603624 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:10.603624 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:10.603624 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:10.603624 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:10.603624 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:10.603624 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:10.603624 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:10.604473 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:10.604473 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:10.604473 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:10.608391 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 283/548] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 278 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:42:10.609806 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-704a1123-94e7-4762-8a84-c3188ca6fc88 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:10.726249 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:10.750050 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-704a1123-94e7-4762-8a84-c3188ca6fc88 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:10.787598 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9591d0ab-5041-4e20-82c2-e815838fb0ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:10.788473 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9591d0ab-5041-4e20-82c2-e815838fb0ae None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:10.789022 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9591d0ab-5041-4e20-82c2-e815838fb0ae None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:10.789468 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9591d0ab-5041-4e20-82c2-e815838fb0ae None admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:10.790305 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9591d0ab-5041-4e20-82c2-e815838fb0ae None admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:10.790748 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9591d0ab-5041-4e20-82c2-e815838fb0ae None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:10.792314 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9591d0ab-5041-4e20-82c2-e815838fb0ae None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:10.805662 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 266/549] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:10.806543 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d5ecad-5c11-4804-a625-cb161dbb05a6 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:10.815905 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9591d0ab-5041-4e20-82c2-e815838fb0ae None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:10.897343 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:10.918447 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d5ecad-5c11-4804-a625-cb161dbb05a6 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:10.954008 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b43622b-44d4-4687-92e3-c1fca3985a10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:10.955985 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b43622b-44d4-4687-92e3-c1fca3985a10 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:10.956243 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b43622b-44d4-4687-92e3-c1fca3985a10 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:10.956761 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b43622b-44d4-4687-92e3-c1fca3985a10 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:10.967981 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b43622b-44d4-4687-92e3-c1fca3985a10 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:42:10.973510 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-4b43622b-44d4-4687-92e3-c1fca3985a10 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 25 00:42:10.973510 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:10.973510 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:10.973510 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:10.973510 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:10.973510 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:10.973510 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:10.973510 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:10.973510 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:10.973510 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:10.973510 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:10.973510 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:10.973510 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:42:10.973510 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:42:10.973510 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:42:10.973510 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:42:10.974609 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:42:10.974609 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:42:10.974609 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:42:10.974609 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:42:10.974609 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:10.974609 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:10.974609 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:10.974609 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:10.974609 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:10.974609 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:10.974609 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:10.974609 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:10.974609 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:10.974609 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:10.974609 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:10.974609 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:10.975427 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:10.975427 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:10.975427 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:10.975427 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:10.975427 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:42:10.975427 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:42:10.975427 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:42:10.975427 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:42:10.975427 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:42:10.975427 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:42:10.975427 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 25 00:42:10.975427 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:10.979662 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9591d0ab-5041-4e20-82c2-e815838fb0ae None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:10.980472 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 284/550] 10.222.0.52 () {56 vars in 1102 bytes} [Sun Sep 25 00:42:10 2022] GET /identity/v3/roles/service => generated 85 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:10.985291 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b43622b-44d4-4687-92e3-c1fca3985a10 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:11.052138 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e92172ce-54a7-4d3a-9a48-38228d03cbe4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:11.053125 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e92172ce-54a7-4d3a-9a48-38228d03cbe4 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:11.053601 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e92172ce-54a7-4d3a-9a48-38228d03cbe4 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:11.054045 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e92172ce-54a7-4d3a-9a48-38228d03cbe4 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:11.065517 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e92172ce-54a7-4d3a-9a48-38228d03cbe4 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:11.074395 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e92172ce-54a7-4d3a-9a48-38228d03cbe4 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:11.093494 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b43622b-44d4-4687-92e3-c1fca3985a10 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:11.094504 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-e92172ce-54a7-4d3a-9a48-38228d03cbe4 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 25 00:42:11.094504 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:11.094504 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:11.094504 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:11.094504 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:11.094504 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:11.094504 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:11.094504 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:11.094504 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:11.094504 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:11.094504 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:11.094504 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:11.094504 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 00:42:11.094504 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 00:42:11.094504 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 00:42:11.094504 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 00:42:11.095405 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:11.095405 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:11.095405 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 00:42:11.095405 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:11.095405 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 00:42:11.095405 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:11.095405 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:11.095405 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:11.095405 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:11.095405 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:11.095405 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:11.095405 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:11.095405 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:11.095405 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:11.095405 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:11.095405 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:11.095405 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:11.095405 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:11.096502 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:11.096502 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:11.096502 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 00:42:11.096502 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 00:42:11.096502 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 00:42:11.096502 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 00:42:11.096502 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 00:42:11.096502 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 00:42:11.096502 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 25 00:42:11.096502 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:11.098868 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 267/551] 10.222.0.52 () {56 vars in 1096 bytes} [Sun Sep 25 00:42:10 2022] GET /identity/v3/users/swift => generated 83 bytes in 284 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:11.110683 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e92172ce-54a7-4d3a-9a48-38228d03cbe4 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:11.115885 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:11.137356 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d699f310-2dbe-4c4a-9bd8-f9dce1f391bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:11.138148 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d699f310-2dbe-4c4a-9bd8-f9dce1f391bd None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:11.138497 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d699f310-2dbe-4c4a-9bd8-f9dce1f391bd None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:11.138862 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d699f310-2dbe-4c4a-9bd8-f9dce1f391bd None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:11.139627 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d699f310-2dbe-4c4a-9bd8-f9dce1f391bd None admin] RBAC: Adding query filter params (name=service) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:11.140038 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d699f310-2dbe-4c4a-9bd8-f9dce1f391bd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:11.149305 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d699f310-2dbe-4c4a-9bd8-f9dce1f391bd None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:11.155373 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 285/552] 10.222.0.52 () {56 vars in 1103 bytes} [Sun Sep 25 00:42:10 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:42:11.156698 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d699f310-2dbe-4c4a-9bd8-f9dce1f391bd None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:11.159843 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:11.266183 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d699f310-2dbe-4c4a-9bd8-f9dce1f391bd None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:11.336595 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4882d5-efb0-4148-bd29-ef9eea88a9a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:11.337509 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4882d5-efb0-4148-bd29-ef9eea88a9a4 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:11.338468 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4882d5-efb0-4148-bd29-ef9eea88a9a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:11.339134 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4882d5-efb0-4148-bd29-ef9eea88a9a4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:11.368114 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e4882d5-efb0-4148-bd29-ef9eea88a9a4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:11.373941 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e4882d5-efb0-4148-bd29-ef9eea88a9a4 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:11.380520 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e92172ce-54a7-4d3a-9a48-38228d03cbe4 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:11.406022 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-9e4882d5-efb0-4148-bd29-ef9eea88a9a4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:11.406022 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:11.406022 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:11.406022 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:11.406022 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:11.406022 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:11.406022 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:11.406022 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:11.406022 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:11.406022 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:11.406022 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:11.406022 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:11.406022 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:11.406022 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:11.406022 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:11.406022 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:11.407104 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:11.407104 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:11.407104 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:11.407104 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:11.407104 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:11.407104 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:11.407104 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:11.407104 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:11.407104 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:11.407104 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:11.407104 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:11.407104 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:11.407104 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:11.407104 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:11.407104 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:11.407104 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:11.407104 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:11.407979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:11.407979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:11.407979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:11.412167 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 286/553] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 251 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:11.413635 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4882d5-efb0-4148-bd29-ef9eea88a9a4 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:11.465878 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea6c2792-d880-40e0-9169-34da0b46c1f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:11.467222 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea6c2792-d880-40e0-9169-34da0b46c1f3 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:11.469423 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea6c2792-d880-40e0-9169-34da0b46c1f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:11.469741 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea6c2792-d880-40e0-9169-34da0b46c1f3 None admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:11.470518 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea6c2792-d880-40e0-9169-34da0b46c1f3 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:11.470830 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea6c2792-d880-40e0-9169-34da0b46c1f3 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:11.472557 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea6c2792-d880-40e0-9169-34da0b46c1f3 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:11.495764 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 268/554] 10.222.0.52 () {56 vars in 1135 bytes} [Sun Sep 25 00:42:11 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 385 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:42:11.501984 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269673e9-f0a9-4822-ac5b-dec65e6238c3 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:11.502686 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269673e9-f0a9-4822-ac5b-dec65e6238c3 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:11.503156 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269673e9-f0a9-4822-ac5b-dec65e6238c3 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:11.505081 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 269/555] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:42:11 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:42:11.506229 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-269673e9-f0a9-4822-ac5b-dec65e6238c3 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:11.510563 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:11.562229 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4882d5-efb0-4148-bd29-ef9eea88a9a4 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:11.599415 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-124397bb-29fa-4de3-8872-ec4c2412d021 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:11.599932 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124397bb-29fa-4de3-8872-ec4c2412d021 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:11.600200 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124397bb-29fa-4de3-8872-ec4c2412d021 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:11.600915 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124397bb-29fa-4de3-8872-ec4c2412d021 None admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:11.601117 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-124397bb-29fa-4de3-8872-ec4c2412d021 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:11.601618 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-124397bb-29fa-4de3-8872-ec4c2412d021 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:11.602976 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-124397bb-29fa-4de3-8872-ec4c2412d021 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:11.611565 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:11.622542 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 287/556] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:11.622542 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139bf815-565e-4a5a-a7a5-282a58da7772 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:11.622542 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139bf815-565e-4a5a-a7a5-282a58da7772 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:11.622542 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139bf815-565e-4a5a-a7a5-282a58da7772 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:11.639591 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-139bf815-565e-4a5a-a7a5-282a58da7772 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:42:11.640792 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-139bf815-565e-4a5a-a7a5-282a58da7772 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:42:11.662849 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-269673e9-f0a9-4822-ac5b-dec65e6238c3 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:11.703822 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c61fb8-5bdf-4271-afbb-e7d7e4db8d03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:11.704097 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c61fb8-5bdf-4271-afbb-e7d7e4db8d03 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:11.704432 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c61fb8-5bdf-4271-afbb-e7d7e4db8d03 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:11.704580 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c61fb8-5bdf-4271-afbb-e7d7e4db8d03 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:11.725316 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 288/557] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:42:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:42:11.725316 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0d0b031-02e7-4475-8067-b103a3986abc None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:11.725316 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0d0b031-02e7-4475-8067-b103a3986abc None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:11.725316 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0d0b031-02e7-4475-8067-b103a3986abc None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:11.725316 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 289/558] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:42:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:42:11.725761 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0d0b031-02e7-4475-8067-b103a3986abc None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:11.782876 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:11.812713 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c61fb8-5bdf-4271-afbb-e7d7e4db8d03 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:11.814218 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c61fb8-5bdf-4271-afbb-e7d7e4db8d03 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:11.855166 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0d0b031-02e7-4475-8067-b103a3986abc None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:11.902806 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:11.906997 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e54c9232-3f64-4dba-942f-7fe3145eef6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:11.907905 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54c9232-3f64-4dba-942f-7fe3145eef6b None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:11.908328 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54c9232-3f64-4dba-942f-7fe3145eef6b None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:11.908743 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54c9232-3f64-4dba-942f-7fe3145eef6b None admin] PATH_INFO: `/v3/users/nova` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:11.918905 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e54c9232-3f64-4dba-942f-7fe3145eef6b None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:11.920621 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e54c9232-3f64-4dba-942f-7fe3145eef6b None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:11.937589 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-e54c9232-3f64-4dba-942f-7fe3145eef6b None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 25 00:42:11.937589 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:11.937589 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:11.937589 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:11.937589 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:11.937589 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:11.937589 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:11.937589 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:11.937589 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:11.937589 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:11.937589 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:11.937589 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:11.937589 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 00:42:11.937589 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 00:42:11.937589 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 00:42:11.937589 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 00:42:11.938781 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:11.938781 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:11.938781 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 00:42:11.938781 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:11.938781 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 00:42:11.938781 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:11.938781 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:11.938781 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:11.938781 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:11.938781 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:11.938781 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:11.938781 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:11.938781 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:11.938781 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:11.938781 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:11.938781 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:11.938781 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:11.938781 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:11.939937 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:11.939937 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:11.939937 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 00:42:11.939937 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 00:42:11.939937 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 00:42:11.939937 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 00:42:11.939937 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 00:42:11.939937 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 00:42:11.939937 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 25 00:42:11.939937 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:11.948238 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 290/559] 10.222.0.52 () {56 vars in 1093 bytes} [Sun Sep 25 00:42:11 2022] GET /identity/v3/users/nova => generated 82 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:11.948238 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4f7069-2780-4581-a642-d9ec9e9d3923 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:11.948238 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4f7069-2780-4581-a642-d9ec9e9d3923 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:11.948238 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4f7069-2780-4581-a642-d9ec9e9d3923 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:11.958900 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 270/560] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:42:11 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 452 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 00:42:11.961298 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c61fb8-5bdf-4271-afbb-e7d7e4db8d03 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:11.980076 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-2b4f7069-2780-4581-a642-d9ec9e9d3923 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:42:11.981281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-2b4f7069-2780-4581-a642-d9ec9e9d3923 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:42:12.064092 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 291/561] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:42:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:42:12.064092 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b4f7069-2780-4581-a642-d9ec9e9d3923 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:12.083125 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c61fb8-5bdf-4271-afbb-e7d7e4db8d03 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:12.107570 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:12.127444 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d309e64e-e1ff-4287-9693-b18333752543 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:12.128161 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d309e64e-e1ff-4287-9693-b18333752543 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:12.128440 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d309e64e-e1ff-4287-9693-b18333752543 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:12.128709 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d309e64e-e1ff-4287-9693-b18333752543 None admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:12.129637 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d309e64e-e1ff-4287-9693-b18333752543 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:12.129942 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d309e64e-e1ff-4287-9693-b18333752543 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:12.131364 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d309e64e-e1ff-4287-9693-b18333752543 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:12.155619 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 271/562] 10.222.0.52 () {56 vars in 1133 bytes} [Sun Sep 25 00:42:11 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:12.156271 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d309e64e-e1ff-4287-9693-b18333752543 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:12.176419 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:12.197962 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b4f7069-2780-4581-a642-d9ec9e9d3923 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:12.242769 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a38d787f-1561-4176-a6d2-046464d2e794 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:12.243373 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38d787f-1561-4176-a6d2-046464d2e794 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:12.243649 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38d787f-1561-4176-a6d2-046464d2e794 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:12.243916 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38d787f-1561-4176-a6d2-046464d2e794 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:12.254583 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a38d787f-1561-4176-a6d2-046464d2e794 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:12.256013 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a38d787f-1561-4176-a6d2-046464d2e794 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:12.271283 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-a38d787f-1561-4176-a6d2-046464d2e794 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:12.271283 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:12.271283 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:12.271283 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:12.271283 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:12.271283 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:12.271283 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:12.271283 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:12.271283 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:12.271283 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:12.271283 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:12.271283 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:12.271283 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:12.271283 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:12.271283 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:12.271283 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:12.272265 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:12.272265 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:12.272265 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:12.272265 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:12.272265 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:12.272265 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:12.272265 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:12.272265 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:12.272265 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:12.272265 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:12.272265 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:12.272265 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:12.272265 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:12.272265 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:12.272265 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:12.272265 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:12.272265 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:12.277941 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:12.277941 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:12.277941 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:12.278213 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d309e64e-e1ff-4287-9693-b18333752543 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:12.278289 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 292/563] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:12.278289 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a38d787f-1561-4176-a6d2-046464d2e794 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:12.320877 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e6b8fb-64e6-4cca-a64a-a882016c244e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:12.321540 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e6b8fb-64e6-4cca-a64a-a882016c244e None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:12.322286 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e6b8fb-64e6-4cca-a64a-a882016c244e None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:12.322559 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e6b8fb-64e6-4cca-a64a-a882016c244e None admin] PATH_INFO: `/v3/roles/service` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:12.338353 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e6b8fb-64e6-4cca-a64a-a882016c244e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:42:12.343276 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-d6e6b8fb-64e6-4cca-a64a-a882016c244e None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 25 00:42:12.343276 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:12.343276 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:12.343276 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:12.343276 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:12.343276 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:12.343276 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:12.343276 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:12.343276 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:12.343276 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:12.343276 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:12.343276 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:12.343276 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:42:12.343276 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:42:12.343276 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:42:12.343276 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:42:12.344237 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:42:12.344237 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:42:12.344237 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:42:12.344237 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:42:12.344237 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:12.344237 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:12.344237 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:12.344237 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:12.344237 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:12.344237 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:12.344237 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:12.344237 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:12.344237 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:12.344237 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:12.344237 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:12.344237 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:12.345089 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:12.345089 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:12.345089 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:12.345089 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:12.345089 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:42:12.345089 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:42:12.345089 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:42:12.345089 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:42:12.345089 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:42:12.345089 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:42:12.345089 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 25 00:42:12.345089 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:12.345089 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 272/564] 10.222.0.52 () {56 vars in 1102 bytes} [Sun Sep 25 00:42:12 2022] GET /identity/v3/roles/service => generated 85 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:12.346373 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073f30b3-a5e7-48b8-8ddf-c9fe377979e4 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:12.346373 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073f30b3-a5e7-48b8-8ddf-c9fe377979e4 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:12.346373 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073f30b3-a5e7-48b8-8ddf-c9fe377979e4 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:12.351387 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 273/565] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:42:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:42:12.351387 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-073f30b3-a5e7-48b8-8ddf-c9fe377979e4 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:12.372989 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:12.389565 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a38d787f-1561-4176-a6d2-046464d2e794 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:12.423960 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64458ab-d066-4f00-b935-59b8a484deae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:12.424582 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64458ab-d066-4f00-b935-59b8a484deae None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:12.424855 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64458ab-d066-4f00-b935-59b8a484deae None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:12.425121 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64458ab-d066-4f00-b935-59b8a484deae None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:12.437121 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e64458ab-d066-4f00-b935-59b8a484deae None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:12.438683 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e64458ab-d066-4f00-b935-59b8a484deae None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:12.439221 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:12.453599 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-e64458ab-d066-4f00-b935-59b8a484deae None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:12.453599 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:12.453599 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:12.453599 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:12.453599 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:12.453599 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:12.453599 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:12.453599 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:12.453599 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:12.453599 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:12.453599 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:12.453599 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:12.453599 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:12.453599 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:12.453599 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:12.453599 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:12.454643 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:12.454643 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:12.454643 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:12.454643 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:12.454643 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:12.454643 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:12.454643 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:12.454643 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:12.454643 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:12.454643 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:12.454643 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:12.454643 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:12.454643 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:12.454643 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:12.454643 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:12.454643 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:12.454643 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:12.455519 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:12.455519 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:12.455519 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:12.455519 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 293/566] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:12.455519 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64458ab-d066-4f00-b935-59b8a484deae None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:12.469779 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-073f30b3-a5e7-48b8-8ddf-c9fe377979e4 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:12.507284 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f47e6128-dd9d-4c5f-ae16-95b6ace219ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:12.508441 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47e6128-dd9d-4c5f-ae16-95b6ace219ab None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:12.508716 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47e6128-dd9d-4c5f-ae16-95b6ace219ab None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:12.508989 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47e6128-dd9d-4c5f-ae16-95b6ace219ab None admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:12.509717 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f47e6128-dd9d-4c5f-ae16-95b6ace219ab None admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:12.510016 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f47e6128-dd9d-4c5f-ae16-95b6ace219ab None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:12.511367 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f47e6128-dd9d-4c5f-ae16-95b6ace219ab None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:12.522906 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 274/567] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:12.525089 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1e3ede-ac79-43e5-9bb4-8354b5d1b9c1 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:12.525395 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1e3ede-ac79-43e5-9bb4-8354b5d1b9c1 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:12.525664 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1e3ede-ac79-43e5-9bb4-8354b5d1b9c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:12.543566 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-8a1e3ede-ac79-43e5-9bb4-8354b5d1b9c1 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:42:12.544754 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-8a1e3ede-ac79-43e5-9bb4-8354b5d1b9c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:42:12.563313 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64458ab-d066-4f00-b935-59b8a484deae None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:12.597903 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de0e082-f54c-4ff6-a140-a6c689f53ed7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:12.598611 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de0e082-f54c-4ff6-a140-a6c689f53ed7 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:12.598886 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de0e082-f54c-4ff6-a140-a6c689f53ed7 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:12.599152 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de0e082-f54c-4ff6-a140-a6c689f53ed7 None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:12.599809 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3de0e082-f54c-4ff6-a140-a6c689f53ed7 None admin] RBAC: Adding query filter params (name=service) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:12.600102 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3de0e082-f54c-4ff6-a140-a6c689f53ed7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:12.601616 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3de0e082-f54c-4ff6-a140-a6c689f53ed7 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:12.606565 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:12.610641 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 294/568] 10.222.0.52 () {56 vars in 1103 bytes} [Sun Sep 25 00:42:12 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:12.611687 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de0e082-f54c-4ff6-a140-a6c689f53ed7 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:12.613470 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 275/569] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:42:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:42:12.614350 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a1e3ede-ac79-43e5-9bb4-8354b5d1b9c1 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:12.697641 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:12.733482 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de0e082-f54c-4ff6-a140-a6c689f53ed7 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:12.733900 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a1e3ede-ac79-43e5-9bb4-8354b5d1b9c1 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:12.772168 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01fa56ac-83aa-4c80-a2ec-2c4941880901 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:12.772420 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e75cc2-5c56-48bb-8d33-f6286e76212c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:12.772794 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01fa56ac-83aa-4c80-a2ec-2c4941880901 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:12.772899 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e75cc2-5c56-48bb-8d33-f6286e76212c None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:12.773122 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01fa56ac-83aa-4c80-a2ec-2c4941880901 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:12.773281 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e75cc2-5c56-48bb-8d33-f6286e76212c None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:12.773497 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01fa56ac-83aa-4c80-a2ec-2c4941880901 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:12.773646 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e75cc2-5c56-48bb-8d33-f6286e76212c None admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:12.774606 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68e75cc2-5c56-48bb-8d33-f6286e76212c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:12.774907 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68e75cc2-5c56-48bb-8d33-f6286e76212c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:12.776666 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68e75cc2-5c56-48bb-8d33-f6286e76212c None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:12.786364 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01fa56ac-83aa-4c80-a2ec-2c4941880901 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:42:12.788936 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 276/570] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:12.790221 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e75cc2-5c56-48bb-8d33-f6286e76212c None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:12.792115 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-01fa56ac-83aa-4c80-a2ec-2c4941880901 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 25 00:42:12.792115 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:12.792115 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:12.792115 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:12.792115 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:12.792115 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:12.792115 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:12.792115 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:12.792115 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:12.792115 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:12.792115 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:12.792115 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:12.792115 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:42:12.792115 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:42:12.792115 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:42:12.792115 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:42:12.793236 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:42:12.793236 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:42:12.793236 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:42:12.793236 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:42:12.793236 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:12.793236 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:12.793236 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:12.793236 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:12.793236 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:12.793236 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:12.793236 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:12.793236 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:12.793236 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:12.793236 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:12.793236 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:12.793236 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:12.794308 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:12.794308 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:12.794308 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:12.794308 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:12.794308 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:42:12.794308 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:42:12.794308 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:42:12.794308 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:42:12.794308 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:42:12.794308 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:42:12.794308 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 25 00:42:12.794308 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:12.795073 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 295/571] 10.222.0.52 () {56 vars in 1102 bytes} [Sun Sep 25 00:42:12 2022] GET /identity/v3/roles/service => generated 85 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:12.795765 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01fa56ac-83aa-4c80-a2ec-2c4941880901 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:12.867787 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:12.907344 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e75cc2-5c56-48bb-8d33-f6286e76212c None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:12.913360 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01fa56ac-83aa-4c80-a2ec-2c4941880901 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:12.935023 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:12.943182 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58362ed7-0236-4436-8452-a48bb7dc59e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:12.943801 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58362ed7-0236-4436-8452-a48bb7dc59e2 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:12.944071 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58362ed7-0236-4436-8452-a48bb7dc59e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:12.944333 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58362ed7-0236-4436-8452-a48bb7dc59e2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:12.949482 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273db896-2baf-4c1f-9d0a-e9ab95d1e4d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:12.950041 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273db896-2baf-4c1f-9d0a-e9ab95d1e4d5 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:12.950309 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273db896-2baf-4c1f-9d0a-e9ab95d1e4d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:12.950571 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273db896-2baf-4c1f-9d0a-e9ab95d1e4d5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:12.951573 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58362ed7-0236-4436-8452-a48bb7dc59e2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:12.953013 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58362ed7-0236-4436-8452-a48bb7dc59e2 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:12.961978 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273db896-2baf-4c1f-9d0a-e9ab95d1e4d5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:12.963392 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273db896-2baf-4c1f-9d0a-e9ab95d1e4d5 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:12.965660 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-58362ed7-0236-4436-8452-a48bb7dc59e2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:42:12.965660 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:12.965660 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:12.965660 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:12.965660 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:12.965660 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:12.965660 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:12.965660 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:12.965660 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:12.965660 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:12.965660 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:12.965660 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:12.965660 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 00:42:12.965660 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 00:42:12.965660 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 00:42:12.966698 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 00:42:12.966698 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:12.966698 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:12.966698 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:12.966698 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:12.966698 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:12.966698 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:12.966698 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:12.966698 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:12.966698 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:12.966698 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:12.966698 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:12.966698 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:12.966698 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:12.966698 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:12.966698 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:12.966698 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:12.967847 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 00:42:12.967847 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 00:42:12.967847 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 00:42:12.967847 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 00:42:12.967847 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 00:42:12.967847 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 00:42:12.967847 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:42:12.967847 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:12.967847 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 277/572] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:42:12 2022] GET /identity/v3/projects/service => generated 88 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:12.967847 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58362ed7-0236-4436-8452-a48bb7dc59e2 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:12.981954 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-273db896-2baf-4c1f-9d0a-e9ab95d1e4d5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:12.981954 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:12.981954 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:12.981954 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:12.981954 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:12.981954 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:12.981954 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:12.981954 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:12.981954 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:12.981954 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:12.981954 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:12.981954 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:12.981954 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:12.981954 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:12.981954 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:12.981954 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:12.983023 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:12.983023 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:12.983023 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:12.983023 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:12.983023 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:12.983023 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:12.983023 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:12.983023 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:12.983023 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:12.983023 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:12.983023 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:12.983023 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:12.983023 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:12.983023 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:12.983023 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:12.983023 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:12.983023 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:12.983924 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:12.983924 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:12.983924 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:12.983924 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 296/573] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:12.984164 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273db896-2baf-4c1f-9d0a-e9ab95d1e4d5 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:13.076853 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58362ed7-0236-4436-8452-a48bb7dc59e2 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:13.084648 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273db896-2baf-4c1f-9d0a-e9ab95d1e4d5 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:13.099043 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:13.111542 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2045267-67b3-4f54-8830-de37374e788a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:13.112167 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2045267-67b3-4f54-8830-de37374e788a None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:13.112437 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2045267-67b3-4f54-8830-de37374e788a None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:13.112709 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2045267-67b3-4f54-8830-de37374e788a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:13.120512 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2045267-67b3-4f54-8830-de37374e788a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:13.121965 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2045267-67b3-4f54-8830-de37374e788a None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:13.122719 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ff47a5-e815-4b81-8c57-b43d6feec512 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:13.123323 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ff47a5-e815-4b81-8c57-b43d6feec512 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:13.123600 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ff47a5-e815-4b81-8c57-b43d6feec512 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:13.123860 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ff47a5-e815-4b81-8c57-b43d6feec512 None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:13.124510 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1ff47a5-e815-4b81-8c57-b43d6feec512 None admin] RBAC: Adding query filter params (name=service) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:13.124804 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1ff47a5-e815-4b81-8c57-b43d6feec512 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:13.126161 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1ff47a5-e815-4b81-8c57-b43d6feec512 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:13.134040 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-b2045267-67b3-4f54-8830-de37374e788a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:42:13.134040 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:13.134040 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:13.134040 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:13.134040 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:13.134040 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:13.134040 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:13.134040 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:13.134040 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:13.134040 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:13.134040 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:13.134040 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:13.134040 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 00:42:13.134040 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 00:42:13.134040 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 00:42:13.134971 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 00:42:13.134971 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:13.134971 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:13.134971 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:13.134971 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:13.134971 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:13.134971 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:13.134971 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:13.134971 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:13.134971 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:13.134971 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:13.134971 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:13.134971 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:13.134971 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:13.134971 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:13.134971 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:13.134971 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:13.135893 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 00:42:13.135893 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 00:42:13.135893 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 00:42:13.135893 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 00:42:13.135893 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 00:42:13.135893 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 00:42:13.135893 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:42:13.135893 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:13.136332 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 297/574] 10.222.0.52 () {56 vars in 1103 bytes} [Sun Sep 25 00:42:12 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:13.136402 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 278/575] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:42:12 2022] GET /identity/v3/projects/service => generated 88 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:13.136402 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2045267-67b3-4f54-8830-de37374e788a None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:13.140443 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ff47a5-e815-4b81-8c57-b43d6feec512 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:13.189488 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:13.241443 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2045267-67b3-4f54-8830-de37374e788a None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:13.242484 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ff47a5-e815-4b81-8c57-b43d6feec512 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:13.277356 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1f8be30-b45a-466a-9c26-1bfe17e6da77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:13.277954 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f8be30-b45a-466a-9c26-1bfe17e6da77 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:13.278082 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc5de5a-b251-4957-9385-9c1b336a4fbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:13.278227 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f8be30-b45a-466a-9c26-1bfe17e6da77 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:13.278488 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f8be30-b45a-466a-9c26-1bfe17e6da77 None admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:13.278591 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc5de5a-b251-4957-9385-9c1b336a4fbb None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:13.278852 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc5de5a-b251-4957-9385-9c1b336a4fbb None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:13.279120 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc5de5a-b251-4957-9385-9c1b336a4fbb None admin] PATH_INFO: `/v3/roles/service` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:13.279210 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1f8be30-b45a-466a-9c26-1bfe17e6da77 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:13.279453 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1f8be30-b45a-466a-9c26-1bfe17e6da77 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:13.280889 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1f8be30-b45a-466a-9c26-1bfe17e6da77 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:13.290670 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dc5de5a-b251-4957-9385-9c1b336a4fbb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:42:13.294279 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 279/576] 10.222.0.52 () {56 vars in 1148 bytes} [Sun Sep 25 00:42:13 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:13.295111 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1f8be30-b45a-466a-9c26-1bfe17e6da77 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:13.296071 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-7dc5de5a-b251-4957-9385-9c1b336a4fbb None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 25 00:42:13.296071 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:13.296071 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:13.296071 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:13.296071 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:13.296071 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:13.296071 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:13.296071 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:13.296071 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:13.296071 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:13.296071 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:13.296071 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:13.296071 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:42:13.296071 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:42:13.296071 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:42:13.296071 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:42:13.296964 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:42:13.296964 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:42:13.296964 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:42:13.296964 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:42:13.296964 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:13.296964 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:13.296964 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:13.296964 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:13.296964 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:13.296964 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:13.296964 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:13.296964 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:13.296964 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:13.296964 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:13.296964 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:13.296964 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:13.297847 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:13.297847 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:13.297847 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:13.297847 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:13.297847 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:42:13.297847 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:42:13.297847 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:42:13.297847 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:42:13.297847 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:42:13.297847 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:42:13.297847 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 25 00:42:13.297847 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:13.297847 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 298/577] 10.222.0.52 () {56 vars in 1102 bytes} [Sun Sep 25 00:42:13 2022] GET /identity/v3/roles/service => generated 85 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:13.298534 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc5de5a-b251-4957-9385-9c1b336a4fbb None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:13.334887 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:13.407872 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1f8be30-b45a-466a-9c26-1bfe17e6da77 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:13.410437 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc5de5a-b251-4957-9385-9c1b336a4fbb None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:13.418314 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:13.444356 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-685915b5-553f-41e3-bc80-4a3c07ccd0b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:13.445572 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685915b5-553f-41e3-bc80-4a3c07ccd0b2 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:13.445937 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685915b5-553f-41e3-bc80-4a3c07ccd0b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:13.446357 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685915b5-553f-41e3-bc80-4a3c07ccd0b2 None admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:13.446455 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c686c94b-9f63-4b61-960f-3775032f034f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:13.446992 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c686c94b-9f63-4b61-960f-3775032f034f None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:13.447096 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-685915b5-553f-41e3-bc80-4a3c07ccd0b2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:13.447268 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c686c94b-9f63-4b61-960f-3775032f034f None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:13.447364 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-685915b5-553f-41e3-bc80-4a3c07ccd0b2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:13.447538 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c686c94b-9f63-4b61-960f-3775032f034f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:13.448637 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-685915b5-553f-41e3-bc80-4a3c07ccd0b2 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:13.459564 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c686c94b-9f63-4b61-960f-3775032f034f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:13.460181 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 280/578] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:13.461093 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-685915b5-553f-41e3-bc80-4a3c07ccd0b2 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:13.461203 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c686c94b-9f63-4b61-960f-3775032f034f None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:13.481294 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-c686c94b-9f63-4b61-960f-3775032f034f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:13.481294 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:13.481294 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:13.481294 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:13.481294 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:13.481294 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:13.481294 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:13.481294 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:13.481294 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:13.481294 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:13.481294 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:13.481294 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:13.481294 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:13.481294 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:13.481294 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:13.481294 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:13.482413 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:13.482413 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:13.482413 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:13.482413 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:13.482413 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:13.482413 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:13.482413 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:13.482413 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:13.482413 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:13.482413 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:13.482413 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:13.482413 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:13.482413 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:13.482413 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:13.482413 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:13.482413 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:13.482413 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:13.484080 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:13.484080 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:13.484080 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:13.484080 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 299/579] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:13.484080 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c686c94b-9f63-4b61-960f-3775032f034f None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:13.573707 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:13.574659 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-685915b5-553f-41e3-bc80-4a3c07ccd0b2 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:13.589417 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c686c94b-9f63-4b61-960f-3775032f034f None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:13.609548 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8432e22e-3bb3-4d51-bf43-69e29a7939c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:13.610155 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8432e22e-3bb3-4d51-bf43-69e29a7939c6 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:13.610431 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8432e22e-3bb3-4d51-bf43-69e29a7939c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:13.610698 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8432e22e-3bb3-4d51-bf43-69e29a7939c6 None admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:13.611380 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8432e22e-3bb3-4d51-bf43-69e29a7939c6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:13.611687 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8432e22e-3bb3-4d51-bf43-69e29a7939c6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:13.613385 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8432e22e-3bb3-4d51-bf43-69e29a7939c6 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:13.625788 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef5b499-08fd-46d2-957b-780932bf219d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:13.626374 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef5b499-08fd-46d2-957b-780932bf219d None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:13.626585 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 281/580] 10.222.0.52 () {56 vars in 1148 bytes} [Sun Sep 25 00:42:13 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:13.626693 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef5b499-08fd-46d2-957b-780932bf219d None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:13.626894 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef5b499-08fd-46d2-957b-780932bf219d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:13.627412 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8432e22e-3bb3-4d51-bf43-69e29a7939c6 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:13.627606 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bef5b499-08fd-46d2-957b-780932bf219d None admin] RBAC: Adding query filter params (role.id=a8e34e460e8a4ed7bcaa54da3c2d5a0c, scope.project.id=9db0fd07fc494203b2ada13f3cd046d3, user.id=da606b3906dd4171bd76fd281e03e258) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:13.627905 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bef5b499-08fd-46d2-957b-780932bf219d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:13.629380 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bef5b499-08fd-46d2-957b-780932bf219d None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:13.637059 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 300/581] 10.222.0.52 () {56 vars in 1374 bytes} [Sun Sep 25 00:42:13 2022] GET /identity/v3/role_assignments?user.id=da606b3906dd4171bd76fd281e03e258&scope.project.id=9db0fd07fc494203b2ada13f3cd046d3&role.id=a8e34e460e8a4ed7bcaa54da3c2d5a0c => generated 259 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:13.638785 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef5b499-08fd-46d2-957b-780932bf219d None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:13.658696 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:13.739129 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8432e22e-3bb3-4d51-bf43-69e29a7939c6 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:13.767395 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef5b499-08fd-46d2-957b-780932bf219d None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:13.782975 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4cc37c1-bb6e-442c-ba4c-c50a4af2a1bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:13.783678 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4cc37c1-bb6e-442c-ba4c-c50a4af2a1bc None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:13.783879 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4cc37c1-bb6e-442c-ba4c-c50a4af2a1bc None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:13.784147 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4cc37c1-bb6e-442c-ba4c-c50a4af2a1bc None admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:13.785425 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4cc37c1-bb6e-442c-ba4c-c50a4af2a1bc None admin] RBAC: Adding query filter params (name=service) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:13.785847 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4cc37c1-bb6e-442c-ba4c-c50a4af2a1bc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:13.787406 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4cc37c1-bb6e-442c-ba4c-c50a4af2a1bc None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:13.813862 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 282/582] 10.222.0.52 () {56 vars in 1103 bytes} [Sun Sep 25 00:42:13 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:13.813862 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4cc37c1-bb6e-442c-ba4c-c50a4af2a1bc None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:13.824361 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc06589-90f7-4ebd-b301-72784e9f7600 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:13.825485 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc06589-90f7-4ebd-b301-72784e9f7600 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:13.826378 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc06589-90f7-4ebd-b301-72784e9f7600 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:13.826378 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc06589-90f7-4ebd-b301-72784e9f7600 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:13.835552 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdc06589-90f7-4ebd-b301-72784e9f7600 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:13.837404 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdc06589-90f7-4ebd-b301-72784e9f7600 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:13.853624 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-cdc06589-90f7-4ebd-b301-72784e9f7600 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 25 00:42:13.853624 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:13.853624 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:13.853624 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:13.853624 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:13.853624 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:13.853624 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:13.853624 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:13.853624 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:13.853624 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:13.853624 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:13.853624 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:13.853624 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 00:42:13.853624 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 00:42:13.853624 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 00:42:13.853624 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 00:42:13.854641 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:13.854641 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:13.854641 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 00:42:13.854641 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:13.854641 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 00:42:13.854641 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:13.854641 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:13.854641 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:13.854641 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:13.854641 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:13.854641 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:13.854641 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:13.854641 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:13.854641 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:13.854641 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:13.854641 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:13.854641 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:13.854641 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:13.855986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:13.855986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:13.855986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 00:42:13.855986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 00:42:13.855986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 00:42:13.855986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 00:42:13.855986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 00:42:13.855986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 00:42:13.855986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 25 00:42:13.855986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:13.855986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 301/583] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:42:13 2022] GET /identity/v3/users/glance => generated 84 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:13.855986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc06589-90f7-4ebd-b301-72784e9f7600 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:13.872627 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:13.919837 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:13.930368 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4cc37c1-bb6e-442c-ba4c-c50a4af2a1bc None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:13.970703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92154797-bc0b-4446-a0ff-71a0a79ce8eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:13.971312 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92154797-bc0b-4446-a0ff-71a0a79ce8eb None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:13.971585 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92154797-bc0b-4446-a0ff-71a0a79ce8eb None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:13.972092 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92154797-bc0b-4446-a0ff-71a0a79ce8eb None admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:13.972482 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92154797-bc0b-4446-a0ff-71a0a79ce8eb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:13.972786 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92154797-bc0b-4446-a0ff-71a0a79ce8eb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:13.974188 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92154797-bc0b-4446-a0ff-71a0a79ce8eb None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:13.979457 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc06589-90f7-4ebd-b301-72784e9f7600 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:13.988732 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 283/584] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:13.993923 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92154797-bc0b-4446-a0ff-71a0a79ce8eb None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:14.016827 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077933f6-a169-4967-becd-2f1076cc1f49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:14.017522 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077933f6-a169-4967-becd-2f1076cc1f49 None admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:14.017804 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077933f6-a169-4967-becd-2f1076cc1f49 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:14.018073 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077933f6-a169-4967-becd-2f1076cc1f49 None admin] PATH_INFO: `/v3/projects/9db0fd07fc494203b2ada13f3cd046d3/users/f142988a4dd14a5f96eacea45c046f85/roles/a8e34e460e8a4ed7bcaa54da3c2d5a0c` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:14.034002 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-077933f6-a169-4967-becd-2f1076cc1f49 None admin] RBAC: Authorizing `identity:create_grant(project_id=9db0fd07fc494203b2ada13f3cd046d3, user_id=f142988a4dd14a5f96eacea45c046f85, role_id=a8e34e460e8a4ed7bcaa54da3c2d5a0c)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:14.035455 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-077933f6-a169-4967-becd-2f1076cc1f49 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:14.050105 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 302/585] 10.222.0.52 () {58 vars in 1439 bytes} [Sun Sep 25 00:42:13 2022] PUT /identity/v3/projects/9db0fd07fc494203b2ada13f3cd046d3/users/f142988a4dd14a5f96eacea45c046f85/roles/a8e34e460e8a4ed7bcaa54da3c2d5a0c => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:42:14.051387 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077933f6-a169-4967-becd-2f1076cc1f49 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:14.126325 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:14.133221 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92154797-bc0b-4446-a0ff-71a0a79ce8eb None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:14.178139 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077933f6-a169-4967-becd-2f1076cc1f49 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:14.179745 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf8a5cad-d5c9-4dd3-b266-bb8c4dc06882 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:14.180745 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8a5cad-d5c9-4dd3-b266-bb8c4dc06882 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:14.181185 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8a5cad-d5c9-4dd3-b266-bb8c4dc06882 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:14.181698 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8a5cad-d5c9-4dd3-b266-bb8c4dc06882 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:14.195454 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf8a5cad-d5c9-4dd3-b266-bb8c4dc06882 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:14.197627 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf8a5cad-d5c9-4dd3-b266-bb8c4dc06882 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:14.216273 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-cf8a5cad-d5c9-4dd3-b266-bb8c4dc06882 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:14.216273 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:14.216273 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:14.216273 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:14.216273 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:14.216273 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:14.216273 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:14.216273 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:14.216273 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:14.216273 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:14.216273 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:14.216273 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:14.216273 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:14.216273 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:14.216273 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:14.216273 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:14.217321 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:14.217321 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:14.217321 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:14.217321 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:14.217321 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:14.217321 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:14.217321 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:14.217321 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:14.217321 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:14.217321 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:14.217321 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:14.217321 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:14.217321 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:14.217321 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:14.217321 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:14.217321 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:14.217321 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:14.218226 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:14.218226 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:14.218226 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:14.223833 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 284/586] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:14.223833 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf8a5cad-d5c9-4dd3-b266-bb8c4dc06882 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:14.228561 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:14.229282 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-534efd0e-b906-4e4a-b507-cd7b5c97eaa6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:14.229974 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534efd0e-b906-4e4a-b507-cd7b5c97eaa6 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:14.230245 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534efd0e-b906-4e4a-b507-cd7b5c97eaa6 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:14.230504 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534efd0e-b906-4e4a-b507-cd7b5c97eaa6 None admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:14.231259 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-534efd0e-b906-4e4a-b507-cd7b5c97eaa6 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:14.231557 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-534efd0e-b906-4e4a-b507-cd7b5c97eaa6 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:14.232988 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-534efd0e-b906-4e4a-b507-cd7b5c97eaa6 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:14.259044 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 303/587] 10.222.0.52 () {56 vars in 1137 bytes} [Sun Sep 25 00:42:14 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:14.260573 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-534efd0e-b906-4e4a-b507-cd7b5c97eaa6 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:14.377886 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf8a5cad-d5c9-4dd3-b266-bb8c4dc06882 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:14.408439 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-534efd0e-b906-4e4a-b507-cd7b5c97eaa6 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:14.420582 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b9a081-43db-4a75-ad11-50cc92b47ef3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:14.421226 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b9a081-43db-4a75-ad11-50cc92b47ef3 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:14.421642 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b9a081-43db-4a75-ad11-50cc92b47ef3 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:14.421919 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b9a081-43db-4a75-ad11-50cc92b47ef3 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:14.432714 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:14.433819 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9b9a081-43db-4a75-ad11-50cc92b47ef3 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:14.435567 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9b9a081-43db-4a75-ad11-50cc92b47ef3 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:14.449921 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-a9b9a081-43db-4a75-ad11-50cc92b47ef3 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 25 00:42:14.449921 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:14.449921 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:14.449921 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:14.449921 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:14.449921 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:14.449921 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:14.449921 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:14.449921 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:14.449921 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:14.449921 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:14.449921 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:14.449921 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 00:42:14.449921 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 00:42:14.449921 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 00:42:14.449921 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 00:42:14.451478 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:14.451478 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:14.451478 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 00:42:14.451478 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:14.451478 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 00:42:14.451478 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:14.451478 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:14.451478 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:14.451478 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:14.451478 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:14.451478 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:14.451478 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:14.451478 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:14.451478 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:14.451478 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:14.451478 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:14.451478 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:14.451478 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:14.452435 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:14.452435 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:14.452435 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 00:42:14.452435 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 00:42:14.452435 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 00:42:14.452435 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 00:42:14.452435 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 00:42:14.452435 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 00:42:14.452435 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 25 00:42:14.452435 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:14.452435 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 285/588] 10.222.0.52 () {56 vars in 1102 bytes} [Sun Sep 25 00:42:14 2022] GET /identity/v3/users/neutron => generated 85 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:14.452435 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b9a081-43db-4a75-ad11-50cc92b47ef3 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:14.461993 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a3f5cf-87cd-4eca-ae27-e859fd740725 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:14.462622 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a3f5cf-87cd-4eca-ae27-e859fd740725 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:14.462896 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a3f5cf-87cd-4eca-ae27-e859fd740725 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:14.463163 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a3f5cf-87cd-4eca-ae27-e859fd740725 None admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:14.463810 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3a3f5cf-87cd-4eca-ae27-e859fd740725 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:14.464838 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3a3f5cf-87cd-4eca-ae27-e859fd740725 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:14.466309 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3a3f5cf-87cd-4eca-ae27-e859fd740725 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:14.490215 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 304/589] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:14.492710 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a3f5cf-87cd-4eca-ae27-e859fd740725 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:14.533035 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:14.586812 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b9a081-43db-4a75-ad11-50cc92b47ef3 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:14.626178 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a3f5cf-87cd-4eca-ae27-e859fd740725 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:14.631855 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c07e0cd-c1b7-4e8d-a678-6eaa25625edf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:14.632571 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c07e0cd-c1b7-4e8d-a678-6eaa25625edf None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:14.632922 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c07e0cd-c1b7-4e8d-a678-6eaa25625edf None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:14.633287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c07e0cd-c1b7-4e8d-a678-6eaa25625edf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:14.648845 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c07e0cd-c1b7-4e8d-a678-6eaa25625edf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:14.650445 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c07e0cd-c1b7-4e8d-a678-6eaa25625edf None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:14.668138 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac47227c-6c3b-4ea2-8bea-1e45acf640ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:14.679476 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-9c07e0cd-c1b7-4e8d-a678-6eaa25625edf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:14.679476 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:14.679476 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:14.679476 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:14.679476 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:14.679476 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:14.679476 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:14.679476 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:14.679476 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:14.679476 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:14.679476 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:14.679476 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:14.679476 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:14.679476 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:14.679476 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:14.679476 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:14.680487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:14.680487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:14.680487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:14.680487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:14.680487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:14.680487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:14.680487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:14.680487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:14.680487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:14.680487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:14.680487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:14.680487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:14.680487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:14.680487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:14.680487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:14.680487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:14.680487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:14.681412 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:14.681412 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:14.681412 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:14.681412 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 286/590] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:14.681412 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c07e0cd-c1b7-4e8d-a678-6eaa25625edf None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:14.682457 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac47227c-6c3b-4ea2-8bea-1e45acf640ff None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:14.683016 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac47227c-6c3b-4ea2-8bea-1e45acf640ff None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:14.683495 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac47227c-6c3b-4ea2-8bea-1e45acf640ff None admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:14.684525 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac47227c-6c3b-4ea2-8bea-1e45acf640ff None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:14.685136 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac47227c-6c3b-4ea2-8bea-1e45acf640ff None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:14.686720 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac47227c-6c3b-4ea2-8bea-1e45acf640ff None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:14.708240 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 305/591] 10.222.0.52 () {56 vars in 1139 bytes} [Sun Sep 25 00:42:14 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:14.709185 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac47227c-6c3b-4ea2-8bea-1e45acf640ff None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:14.755449 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:14.810076 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c07e0cd-c1b7-4e8d-a678-6eaa25625edf None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:14.829774 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:14.841167 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac47227c-6c3b-4ea2-8bea-1e45acf640ff None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:14.856539 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ac9176-986a-4b56-bad4-a146fda3bfc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:14.857343 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ac9176-986a-4b56-bad4-a146fda3bfc2 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:14.857719 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ac9176-986a-4b56-bad4-a146fda3bfc2 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:14.858090 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ac9176-986a-4b56-bad4-a146fda3bfc2 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:14.870028 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1ac9176-986a-4b56-bad4-a146fda3bfc2 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:14.871468 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1ac9176-986a-4b56-bad4-a146fda3bfc2 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:14.888163 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-c1ac9176-986a-4b56-bad4-a146fda3bfc2 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 25 00:42:14.888163 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:14.888163 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:14.888163 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:14.888163 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:14.888163 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:14.888163 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:14.888163 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:14.888163 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:14.888163 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:14.888163 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:14.888163 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:14.888163 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 00:42:14.888163 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 00:42:14.888163 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 00:42:14.888163 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 00:42:14.889146 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:14.889146 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:14.889146 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 00:42:14.889146 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:14.889146 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 00:42:14.889146 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:14.889146 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:14.889146 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:14.889146 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:14.889146 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:14.889146 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:14.889146 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:14.889146 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:14.889146 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:14.889146 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:14.889146 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:14.889146 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:14.889146 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:14.890623 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:14.890623 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:14.890623 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 00:42:14.890623 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 00:42:14.890623 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 00:42:14.890623 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 00:42:14.890623 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 00:42:14.890623 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 00:42:14.890623 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 25 00:42:14.890623 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:14.893672 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1028b869-565f-4033-8b88-e6a73c9fdced None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:14.893958 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 287/592] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:42:14 2022] GET /identity/v3/users/cinder => generated 84 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:14.894760 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ac9176-986a-4b56-bad4-a146fda3bfc2 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:14.894968 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1028b869-565f-4033-8b88-e6a73c9fdced None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:14.895450 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1028b869-565f-4033-8b88-e6a73c9fdced None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:14.895912 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1028b869-565f-4033-8b88-e6a73c9fdced None admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:14.896773 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1028b869-565f-4033-8b88-e6a73c9fdced None admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:14.897345 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1028b869-565f-4033-8b88-e6a73c9fdced None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:14.900971 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1028b869-565f-4033-8b88-e6a73c9fdced None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:14.916702 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 306/593] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:14.917381 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1028b869-565f-4033-8b88-e6a73c9fdced None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:15.033158 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ac9176-986a-4b56-bad4-a146fda3bfc2 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:15.033427 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:15.042740 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1028b869-565f-4033-8b88-e6a73c9fdced None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:15.072511 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560d0aa0-ffd9-45d1-ba34-a62808c37fcd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:15.073165 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560d0aa0-ffd9-45d1-ba34-a62808c37fcd None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:15.073491 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560d0aa0-ffd9-45d1-ba34-a62808c37fcd None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:15.073760 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560d0aa0-ffd9-45d1-ba34-a62808c37fcd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:15.086739 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-560d0aa0-ffd9-45d1-ba34-a62808c37fcd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:15.088227 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-560d0aa0-ffd9-45d1-ba34-a62808c37fcd None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:15.093921 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3660a9db-0bed-4b82-b681-4a26e2e91af8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:15.094707 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3660a9db-0bed-4b82-b681-4a26e2e91af8 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:15.095617 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3660a9db-0bed-4b82-b681-4a26e2e91af8 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:15.096051 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3660a9db-0bed-4b82-b681-4a26e2e91af8 None admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:15.097456 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3660a9db-0bed-4b82-b681-4a26e2e91af8 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:15.098199 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3660a9db-0bed-4b82-b681-4a26e2e91af8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:15.099825 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3660a9db-0bed-4b82-b681-4a26e2e91af8 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:15.103655 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-560d0aa0-ffd9-45d1-ba34-a62808c37fcd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:15.103655 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:15.103655 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:15.103655 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:15.103655 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:15.103655 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:15.103655 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:15.103655 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:15.103655 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:15.103655 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:15.103655 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:15.103655 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:15.103655 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:15.103655 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:15.103655 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:15.103655 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:15.104817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:15.104817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:15.104817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:15.104817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:15.104817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:15.104817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:15.104817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:15.104817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:15.104817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:15.104817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:15.104817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:15.104817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:15.104817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:15.104817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:15.104817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:15.104817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:15.104817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:15.106114 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:15.106114 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:15.106114 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:15.106114 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 288/594] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:15.106114 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560d0aa0-ffd9-45d1-ba34-a62808c37fcd None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:15.127381 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 307/595] 10.222.0.52 () {56 vars in 1137 bytes} [Sun Sep 25 00:42:14 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:15.128585 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3660a9db-0bed-4b82-b681-4a26e2e91af8 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:15.140684 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:15.220021 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560d0aa0-ffd9-45d1-ba34-a62808c37fcd None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:15.256550 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735d73ba-856d-4978-84c8-bc0d12f763b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:15.257175 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735d73ba-856d-4978-84c8-bc0d12f763b6 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:15.257812 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735d73ba-856d-4978-84c8-bc0d12f763b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:15.257942 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735d73ba-856d-4978-84c8-bc0d12f763b6 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:15.265503 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-735d73ba-856d-4978-84c8-bc0d12f763b6 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:15.267003 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-735d73ba-856d-4978-84c8-bc0d12f763b6 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:15.268544 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3660a9db-0bed-4b82-b681-4a26e2e91af8 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:15.279701 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-735d73ba-856d-4978-84c8-bc0d12f763b6 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:42:15.279701 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:15.279701 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:15.279701 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:15.279701 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:15.279701 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:15.279701 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:15.279701 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:15.279701 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:15.279701 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:15.279701 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:15.279701 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:15.279701 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 00:42:15.279701 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 00:42:15.279701 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 00:42:15.280820 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 00:42:15.280820 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:15.280820 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:15.280820 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:15.280820 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:15.280820 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:15.280820 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:15.280820 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:15.280820 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:15.280820 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:15.280820 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:15.280820 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:15.280820 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:15.280820 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:15.280820 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:15.280820 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:15.280820 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:15.282069 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 00:42:15.282069 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 00:42:15.282069 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 00:42:15.282069 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 00:42:15.282069 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 00:42:15.282069 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 00:42:15.282069 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:42:15.282069 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:15.282069 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 289/596] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:42:15 2022] GET /identity/v3/projects/service => generated 88 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:15.282069 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735d73ba-856d-4978-84c8-bc0d12f763b6 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:15.319717 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d21fda7-b25c-47d8-901d-47aa330aefab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:15.320659 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d21fda7-b25c-47d8-901d-47aa330aefab None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:15.321349 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d21fda7-b25c-47d8-901d-47aa330aefab None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:15.321789 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d21fda7-b25c-47d8-901d-47aa330aefab None admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:15.323386 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d21fda7-b25c-47d8-901d-47aa330aefab None admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:15.323841 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d21fda7-b25c-47d8-901d-47aa330aefab None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:15.325391 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d21fda7-b25c-47d8-901d-47aa330aefab None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:15.341227 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 308/597] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:15.342880 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d21fda7-b25c-47d8-901d-47aa330aefab None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:15.364617 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:15.381322 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:15.398914 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735d73ba-856d-4978-84c8-bc0d12f763b6 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:15.434780 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f959ba-505b-4570-8a3e-3ed3a5b75e4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:15.435459 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f959ba-505b-4570-8a3e-3ed3a5b75e4c None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:15.435741 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f959ba-505b-4570-8a3e-3ed3a5b75e4c None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:15.436012 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f959ba-505b-4570-8a3e-3ed3a5b75e4c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:15.448639 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56f959ba-505b-4570-8a3e-3ed3a5b75e4c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:15.450184 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56f959ba-505b-4570-8a3e-3ed3a5b75e4c None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:15.466699 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-56f959ba-505b-4570-8a3e-3ed3a5b75e4c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:15.466699 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:15.466699 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:15.466699 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:15.466699 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:15.466699 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:15.466699 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:15.466699 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:15.466699 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:15.466699 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:15.466699 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:15.466699 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:15.466699 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:15.466699 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:15.466699 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:15.466699 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:15.467997 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:15.467997 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:15.467997 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:15.467997 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:15.467997 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:15.467997 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:15.467997 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:15.467997 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:15.467997 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:15.467997 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:15.467997 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:15.467997 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:15.467997 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:15.467997 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:15.467997 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:15.467997 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:15.467997 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:15.469094 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:15.469094 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:15.469094 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:15.470326 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 290/598] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:15.471128 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f959ba-505b-4570-8a3e-3ed3a5b75e4c None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:15.504038 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d21fda7-b25c-47d8-901d-47aa330aefab None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:15.554014 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a010eb1e-378f-4849-84e2-7660888c021a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:15.554944 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a010eb1e-378f-4849-84e2-7660888c021a None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:15.555508 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a010eb1e-378f-4849-84e2-7660888c021a None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:15.556045 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a010eb1e-378f-4849-84e2-7660888c021a None admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:15.556901 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a010eb1e-378f-4849-84e2-7660888c021a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:15.557606 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a010eb1e-378f-4849-84e2-7660888c021a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:15.560106 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a010eb1e-378f-4849-84e2-7660888c021a None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:15.577237 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 309/599] 10.222.0.52 () {56 vars in 1148 bytes} [Sun Sep 25 00:42:15 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:15.578397 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a010eb1e-378f-4849-84e2-7660888c021a None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:15.595968 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f959ba-505b-4570-8a3e-3ed3a5b75e4c None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:15.636605 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe701a70-5665-43b0-a36c-4ecfd6c556f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:15.637473 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe701a70-5665-43b0-a36c-4ecfd6c556f7 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:15.637857 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe701a70-5665-43b0-a36c-4ecfd6c556f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:15.638217 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe701a70-5665-43b0-a36c-4ecfd6c556f7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:15.649709 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe701a70-5665-43b0-a36c-4ecfd6c556f7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:15.652366 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe701a70-5665-43b0-a36c-4ecfd6c556f7 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:15.655474 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:15.675967 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:15.676764 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-fe701a70-5665-43b0-a36c-4ecfd6c556f7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:42:15.676764 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:15.676764 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:15.676764 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:15.676764 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:15.676764 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:15.676764 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:15.676764 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:15.676764 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:15.676764 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:15.676764 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:15.676764 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:15.676764 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 00:42:15.676764 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 00:42:15.676764 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 00:42:15.677871 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 00:42:15.677871 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:15.677871 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:15.677871 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:15.677871 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:15.677871 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:15.677871 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:15.677871 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:15.677871 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:15.677871 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:15.677871 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:15.677871 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:15.677871 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:15.677871 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:15.677871 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:15.677871 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:15.677871 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:15.679009 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 00:42:15.679009 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 00:42:15.679009 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 00:42:15.679009 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 00:42:15.679009 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 00:42:15.679009 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 00:42:15.679009 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:42:15.679009 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:15.685228 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 291/600] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:42:15 2022] GET /identity/v3/projects/service => generated 88 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:15.686629 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe701a70-5665-43b0-a36c-4ecfd6c556f7 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:15.719037 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a010eb1e-378f-4849-84e2-7660888c021a None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:15.769771 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25a246e-14c2-4932-837b-b82c033121fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:15.770726 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25a246e-14c2-4932-837b-b82c033121fe None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:15.771147 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25a246e-14c2-4932-837b-b82c033121fe None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:15.771568 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25a246e-14c2-4932-837b-b82c033121fe None admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:15.772380 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b25a246e-14c2-4932-837b-b82c033121fe None admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:15.772830 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b25a246e-14c2-4932-837b-b82c033121fe None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:15.775328 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b25a246e-14c2-4932-837b-b82c033121fe None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:15.792284 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 310/601] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:15.794515 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25a246e-14c2-4932-837b-b82c033121fe None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:15.806995 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe701a70-5665-43b0-a36c-4ecfd6c556f7 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:15.849858 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a629218-c072-47b6-b920-3a9c0b3c9fb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:15.851467 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a629218-c072-47b6-b920-3a9c0b3c9fb7 None admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:15.851467 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a629218-c072-47b6-b920-3a9c0b3c9fb7 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:15.851467 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a629218-c072-47b6-b920-3a9c0b3c9fb7 None admin] PATH_INFO: `/v3/projects/9db0fd07fc494203b2ada13f3cd046d3/users/4ebfc0abf1b842459a44ca9324252688/roles/a8e34e460e8a4ed7bcaa54da3c2d5a0c` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:15.871421 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a629218-c072-47b6-b920-3a9c0b3c9fb7 None admin] RBAC: Authorizing `identity:create_grant(project_id=9db0fd07fc494203b2ada13f3cd046d3, user_id=4ebfc0abf1b842459a44ca9324252688, role_id=a8e34e460e8a4ed7bcaa54da3c2d5a0c)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:15.873009 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a629218-c072-47b6-b920-3a9c0b3c9fb7 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:15.893475 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 292/602] 10.222.0.52 () {58 vars in 1439 bytes} [Sun Sep 25 00:42:15 2022] PUT /identity/v3/projects/9db0fd07fc494203b2ada13f3cd046d3/users/4ebfc0abf1b842459a44ca9324252688/roles/a8e34e460e8a4ed7bcaa54da3c2d5a0c => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:42:15.894670 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a629218-c072-47b6-b920-3a9c0b3c9fb7 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:15.958765 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25a246e-14c2-4932-837b-b82c033121fe None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:15.993315 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:16.010235 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c78b9c6-fa27-48a2-837c-14cc75e343cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:16.010853 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c78b9c6-fa27-48a2-837c-14cc75e343cf None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:16.011126 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c78b9c6-fa27-48a2-837c-14cc75e343cf None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:16.011391 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c78b9c6-fa27-48a2-837c-14cc75e343cf None admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:16.012091 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c78b9c6-fa27-48a2-837c-14cc75e343cf None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:16.012389 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c78b9c6-fa27-48a2-837c-14cc75e343cf None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:16.014675 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c78b9c6-fa27-48a2-837c-14cc75e343cf None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:16.017025 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:16.035820 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 311/603] 10.222.0.52 () {56 vars in 1148 bytes} [Sun Sep 25 00:42:15 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:16.037119 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b2f1a1-fa6e-480c-b6a5-4a4aa9c0f9f7 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:16.037466 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b2f1a1-fa6e-480c-b6a5-4a4aa9c0f9f7 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:16.037737 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b2f1a1-fa6e-480c-b6a5-4a4aa9c0f9f7 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:16.044841 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 312/604] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:42:16 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:42:16.045990 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b2f1a1-fa6e-480c-b6a5-4a4aa9c0f9f7 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:16.078519 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a629218-c072-47b6-b920-3a9c0b3c9fb7 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:16.136256 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1974198-6bf9-42d8-b299-76c09bc228e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:16.137205 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1974198-6bf9-42d8-b299-76c09bc228e7 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:16.137697 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1974198-6bf9-42d8-b299-76c09bc228e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:16.138157 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1974198-6bf9-42d8-b299-76c09bc228e7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:16.145906 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1974198-6bf9-42d8-b299-76c09bc228e7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:16.147362 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1974198-6bf9-42d8-b299-76c09bc228e7 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:16.161727 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-c1974198-6bf9-42d8-b299-76c09bc228e7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:42:16.161727 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:16.161727 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:16.161727 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:16.161727 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:16.161727 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:16.161727 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:16.161727 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:16.161727 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:16.161727 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:16.161727 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:16.161727 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:16.161727 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 00:42:16.161727 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 00:42:16.161727 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 00:42:16.162943 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 00:42:16.162943 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:16.162943 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:16.162943 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:16.162943 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:16.162943 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:16.162943 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:16.162943 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:16.162943 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:16.162943 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:16.162943 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:16.162943 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:16.162943 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:16.162943 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:16.162943 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:16.162943 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:16.162943 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:16.164590 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 00:42:16.164590 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 00:42:16.164590 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 00:42:16.164590 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 00:42:16.164590 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 00:42:16.164590 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 00:42:16.164590 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:42:16.164590 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:16.164590 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 293/605] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:42:15 2022] GET /identity/v3/projects/service => generated 88 bytes in 269 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:16.165093 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd518faf-b41f-4148-8e23-8485244ace84 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:16.165757 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd518faf-b41f-4148-8e23-8485244ace84 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:16.165757 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd518faf-b41f-4148-8e23-8485244ace84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:16.194661 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-bd518faf-b41f-4148-8e23-8485244ace84 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:42:16.196809 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-bd518faf-b41f-4148-8e23-8485244ace84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:42:16.198362 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b2f1a1-fa6e-480c-b6a5-4a4aa9c0f9f7 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:16.243962 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aff2c1f-8acc-4878-9fcc-11dd92578b16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:16.245035 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aff2c1f-8acc-4878-9fcc-11dd92578b16 None admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:16.245571 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aff2c1f-8acc-4878-9fcc-11dd92578b16 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:16.246327 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aff2c1f-8acc-4878-9fcc-11dd92578b16 None admin] PATH_INFO: `/v3/projects/9db0fd07fc494203b2ada13f3cd046d3/users/20c274e96dec478b98ef51f80ea893b0/roles/a8e34e460e8a4ed7bcaa54da3c2d5a0c` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:16.266821 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aff2c1f-8acc-4878-9fcc-11dd92578b16 None admin] RBAC: Authorizing `identity:create_grant(project_id=9db0fd07fc494203b2ada13f3cd046d3, user_id=20c274e96dec478b98ef51f80ea893b0, role_id=a8e34e460e8a4ed7bcaa54da3c2d5a0c)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:16.269318 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aff2c1f-8acc-4878-9fcc-11dd92578b16 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:16.294666 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 313/606] 10.222.0.52 () {58 vars in 1439 bytes} [Sun Sep 25 00:42:16 2022] PUT /identity/v3/projects/9db0fd07fc494203b2ada13f3cd046d3/users/20c274e96dec478b98ef51f80ea893b0/roles/a8e34e460e8a4ed7bcaa54da3c2d5a0c => generated 0 bytes in 244 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:42:16.295819 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aff2c1f-8acc-4878-9fcc-11dd92578b16 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:16.319288 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 294/607] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:42:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:42:16.401879 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:16.504698 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aff2c1f-8acc-4878-9fcc-11dd92578b16 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:16.534890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868dca43-cb44-4751-a99f-8c0acf1cf674 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:16.535417 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868dca43-cb44-4751-a99f-8c0acf1cf674 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:16.535417 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868dca43-cb44-4751-a99f-8c0acf1cf674 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:16.536903 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 295/608] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:42:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:42:16.543176 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e911c7a7-f94e-490e-a45d-c5c201c270e3 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:16.543465 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e911c7a7-f94e-490e-a45d-c5c201c270e3 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:16.544041 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e911c7a7-f94e-490e-a45d-c5c201c270e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:16.561326 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e91fee-cf73-4301-b02f-c18dc99e6ab7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:16.562902 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e91fee-cf73-4301-b02f-c18dc99e6ab7 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:16.563334 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e91fee-cf73-4301-b02f-c18dc99e6ab7 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:16.563839 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e91fee-cf73-4301-b02f-c18dc99e6ab7 None admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:16.565185 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e91fee-cf73-4301-b02f-c18dc99e6ab7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:16.565683 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e91fee-cf73-4301-b02f-c18dc99e6ab7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:16.567298 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e91fee-cf73-4301-b02f-c18dc99e6ab7 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:16.570348 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-e911c7a7-f94e-490e-a45d-c5c201c270e3 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:42:16.572494 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-e911c7a7-f94e-490e-a45d-c5c201c270e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:42:16.583032 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 314/609] 10.222.0.52 () {56 vars in 1148 bytes} [Sun Sep 25 00:42:16 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:16.589617 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e91fee-cf73-4301-b02f-c18dc99e6ab7 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:16.620351 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:16.685437 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 296/610] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:42:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:42:16.724443 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e91fee-cf73-4301-b02f-c18dc99e6ab7 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:16.727437 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:16.764416 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe967a6-9eca-4084-b4d4-7432c3c0cae1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:16.765452 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe967a6-9eca-4084-b4d4-7432c3c0cae1 None admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:16.765880 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe967a6-9eca-4084-b4d4-7432c3c0cae1 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:16.766312 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe967a6-9eca-4084-b4d4-7432c3c0cae1 None admin] PATH_INFO: `/v3/projects/9db0fd07fc494203b2ada13f3cd046d3/users/944f10f38ee2428a85599f6a714ac806/roles/a8e34e460e8a4ed7bcaa54da3c2d5a0c` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:16.786700 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe967a6-9eca-4084-b4d4-7432c3c0cae1 None admin] RBAC: Authorizing `identity:create_grant(project_id=9db0fd07fc494203b2ada13f3cd046d3, user_id=944f10f38ee2428a85599f6a714ac806, role_id=a8e34e460e8a4ed7bcaa54da3c2d5a0c)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:16.788444 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe967a6-9eca-4084-b4d4-7432c3c0cae1 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:16.812038 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 315/611] 10.222.0.52 () {58 vars in 1439 bytes} [Sun Sep 25 00:42:16 2022] PUT /identity/v3/projects/9db0fd07fc494203b2ada13f3cd046d3/users/944f10f38ee2428a85599f6a714ac806/roles/a8e34e460e8a4ed7bcaa54da3c2d5a0c => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 00:42:16.932892 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e911c7a7-f94e-490e-a45d-c5c201c270e3 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:17.080922 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e911c7a7-f94e-490e-a45d-c5c201c270e3 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:17.107023 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe967a6-9eca-4084-b4d4-7432c3c0cae1 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:17.124369 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f7c70b-7880-4ec4-807d-ea3ff46f4ca8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:17.125391 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f7c70b-7880-4ec4-807d-ea3ff46f4ca8 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:17.125840 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f7c70b-7880-4ec4-807d-ea3ff46f4ca8 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:17.126262 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f7c70b-7880-4ec4-807d-ea3ff46f4ca8 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:17.139148 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83f7c70b-7880-4ec4-807d-ea3ff46f4ca8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:42:17.144711 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-83f7c70b-7880-4ec4-807d-ea3ff46f4ca8 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 25 00:42:17.144711 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:17.144711 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:17.144711 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:17.144711 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:17.144711 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:17.144711 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:17.144711 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:17.144711 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:17.144711 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:17.144711 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:17.144711 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:17.144711 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:42:17.144711 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:42:17.144711 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:42:17.144711 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:42:17.145725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:42:17.145725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:42:17.145725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:42:17.145725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:42:17.145725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:17.145725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:17.145725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:17.145725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:17.145725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:17.145725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:17.145725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:17.145725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:17.145725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:17.145725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:17.145725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:17.145725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:17.146594 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:17.146594 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:17.146594 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:17.146594 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:17.146594 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:42:17.146594 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:42:17.146594 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:42:17.146594 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:42:17.146594 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:42:17.146594 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:42:17.146594 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 25 00:42:17.146594 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:17.149060 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 297/612] 10.222.0.52 () {56 vars in 1102 bytes} [Sun Sep 25 00:42:16 2022] GET /identity/v3/roles/service => generated 85 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:17.154609 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f7c70b-7880-4ec4-807d-ea3ff46f4ca8 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:17.217102 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe967a6-9eca-4084-b4d4-7432c3c0cae1 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:17.222489 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:17.270230 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e061814e-a525-4172-a3e0-754e9cfcfbdc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:17.271008 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e061814e-a525-4172-a3e0-754e9cfcfbdc None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:17.271384 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e061814e-a525-4172-a3e0-754e9cfcfbdc None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:17.271774 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e061814e-a525-4172-a3e0-754e9cfcfbdc None admin] PATH_INFO: `/v3/roles/service` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:17.282730 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e061814e-a525-4172-a3e0-754e9cfcfbdc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:42:17.287806 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-e061814e-a525-4172-a3e0-754e9cfcfbdc None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 25 00:42:17.287806 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:17.287806 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:17.287806 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:17.287806 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:17.287806 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:17.287806 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:17.287806 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:17.287806 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:17.287806 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:17.287806 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:17.287806 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:17.287806 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:42:17.287806 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:42:17.287806 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:42:17.287806 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:42:17.288836 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:42:17.288836 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:42:17.288836 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:42:17.288836 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:42:17.288836 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:17.288836 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:17.288836 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:17.288836 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:17.288836 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:17.288836 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:17.288836 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:17.288836 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:17.288836 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:17.288836 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:17.288836 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:17.288836 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:17.289716 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:17.289716 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:17.289716 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:17.289716 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:17.289716 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:42:17.289716 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:42:17.289716 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:42:17.289716 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:42:17.289716 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:42:17.289716 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:42:17.289716 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 25 00:42:17.289716 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:17.289716 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 316/613] 10.222.0.52 () {56 vars in 1102 bytes} [Sun Sep 25 00:42:17 2022] GET /identity/v3/roles/service => generated 85 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:42:17.302370 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e061814e-a525-4172-a3e0-754e9cfcfbdc None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:17.326461 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:17.418709 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e061814e-a525-4172-a3e0-754e9cfcfbdc None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:17.428543 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f7c70b-7880-4ec4-807d-ea3ff46f4ca8 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:17.461055 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-092222e9-600f-486f-8fc1-02ed4bb138bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:17.462170 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092222e9-600f-486f-8fc1-02ed4bb138bb None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:17.462586 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092222e9-600f-486f-8fc1-02ed4bb138bb None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:17.462850 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092222e9-600f-486f-8fc1-02ed4bb138bb None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:17.463521 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-092222e9-600f-486f-8fc1-02ed4bb138bb None admin] RBAC: Adding query filter params (name=service) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:17.463814 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-092222e9-600f-486f-8fc1-02ed4bb138bb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:17.465479 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-092222e9-600f-486f-8fc1-02ed4bb138bb None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:17.478712 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 317/614] 10.222.0.52 () {56 vars in 1103 bytes} [Sun Sep 25 00:42:17 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:42:17.486688 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-092222e9-600f-486f-8fc1-02ed4bb138bb None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:17.515491 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-828a8b05-e017-4cff-8bb1-82f063f70f0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:17.516814 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828a8b05-e017-4cff-8bb1-82f063f70f0e None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:17.517659 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828a8b05-e017-4cff-8bb1-82f063f70f0e None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:17.518376 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828a8b05-e017-4cff-8bb1-82f063f70f0e None admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:17.519716 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-828a8b05-e017-4cff-8bb1-82f063f70f0e None admin] RBAC: Adding query filter params (name=service) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:17.521184 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-828a8b05-e017-4cff-8bb1-82f063f70f0e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:17.525404 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-828a8b05-e017-4cff-8bb1-82f063f70f0e None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:17.548401 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 298/615] 10.222.0.52 () {56 vars in 1103 bytes} [Sun Sep 25 00:42:17 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 394 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:42:17.553962 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-828a8b05-e017-4cff-8bb1-82f063f70f0e None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:17.678619 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:17.679957 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:17.692609 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-828a8b05-e017-4cff-8bb1-82f063f70f0e None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:17.723142 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-092222e9-600f-486f-8fc1-02ed4bb138bb None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:17.763415 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90109b2b-1657-4ea9-987c-306f84207f7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:17.764515 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90109b2b-1657-4ea9-987c-306f84207f7e None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:17.765040 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90109b2b-1657-4ea9-987c-306f84207f7e None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:17.765518 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90109b2b-1657-4ea9-987c-306f84207f7e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:17.778040 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-557b3bb7-802a-4d9d-b570-e5092b1e84ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:17.778650 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90109b2b-1657-4ea9-987c-306f84207f7e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:17.778764 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557b3bb7-802a-4d9d-b570-e5092b1e84ba None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:17.778955 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557b3bb7-802a-4d9d-b570-e5092b1e84ba None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:17.779230 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557b3bb7-802a-4d9d-b570-e5092b1e84ba None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:17.780846 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90109b2b-1657-4ea9-987c-306f84207f7e None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:17.791725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-557b3bb7-802a-4d9d-b570-e5092b1e84ba None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:17.793119 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-557b3bb7-802a-4d9d-b570-e5092b1e84ba None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:17.797400 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-90109b2b-1657-4ea9-987c-306f84207f7e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:17.797400 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:17.797400 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:17.797400 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:17.797400 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:17.797400 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:17.797400 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:17.797400 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:17.797400 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:17.797400 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:17.797400 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:17.797400 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:17.797400 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:17.797400 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:17.797400 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:17.797400 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:17.798363 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:17.798363 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:17.798363 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:17.798363 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:17.798363 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:17.798363 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:17.798363 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:17.798363 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:17.798363 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:17.798363 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:17.798363 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:17.798363 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:17.798363 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:17.798363 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:17.798363 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:17.798363 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:17.798363 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:17.799293 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:17.799293 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:17.799293 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:17.801405 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 318/616] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 315 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:42:17.808120 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90109b2b-1657-4ea9-987c-306f84207f7e None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:17.813657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-557b3bb7-802a-4d9d-b570-e5092b1e84ba None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:17.813657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:17.813657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:17.813657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:17.813657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:17.813657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:17.813657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:17.813657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:17.813657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:17.813657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:17.813657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:17.813657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:17.813657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:17.813657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:17.813657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:17.813657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:17.814618 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:17.814618 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:17.814618 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:17.814618 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:17.814618 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:17.814618 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:17.814618 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:17.814618 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:17.814618 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:17.814618 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:17.814618 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:17.814618 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:17.814618 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:17.814618 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:17.814618 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:17.814618 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:17.814618 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:17.815544 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:17.815544 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:17.815544 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:17.818275 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 299/617] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 264 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:17.825176 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-557b3bb7-802a-4d9d-b570-e5092b1e84ba None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:18.009956 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-557b3bb7-802a-4d9d-b570-e5092b1e84ba None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:18.023271 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90109b2b-1657-4ea9-987c-306f84207f7e None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:18.080225 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:18.088564 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee58f611-a912-4791-bff0-5bf7f8ba0233 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:18.089493 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee58f611-a912-4791-bff0-5bf7f8ba0233 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:18.090837 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee58f611-a912-4791-bff0-5bf7f8ba0233 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:18.091295 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee58f611-a912-4791-bff0-5bf7f8ba0233 None admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:18.092485 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee58f611-a912-4791-bff0-5bf7f8ba0233 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:18.093492 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee58f611-a912-4791-bff0-5bf7f8ba0233 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:18.095520 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee58f611-a912-4791-bff0-5bf7f8ba0233 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:18.099974 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0145fd-5f25-40b3-ad49-861665ff303e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:18.100809 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0145fd-5f25-40b3-ad49-861665ff303e None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:18.101345 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0145fd-5f25-40b3-ad49-861665ff303e None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:18.101896 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0145fd-5f25-40b3-ad49-861665ff303e None admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:18.102718 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c0145fd-5f25-40b3-ad49-861665ff303e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:18.107760 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c0145fd-5f25-40b3-ad49-861665ff303e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:18.109443 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c0145fd-5f25-40b3-ad49-861665ff303e None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:18.119360 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 300/618] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 294 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:18.126406 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee58f611-a912-4791-bff0-5bf7f8ba0233 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:18.129073 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 319/619] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 321 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:42:18.134001 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:18.135548 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0145fd-5f25-40b3-ad49-861665ff303e None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:18.292455 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee58f611-a912-4791-bff0-5bf7f8ba0233 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:18.334921 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0145fd-5f25-40b3-ad49-861665ff303e None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:18.357522 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a5d8f8-cf9e-4c1d-827a-d90bb14bb27e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:18.358154 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a5d8f8-cf9e-4c1d-827a-d90bb14bb27e None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:18.358429 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a5d8f8-cf9e-4c1d-827a-d90bb14bb27e None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:18.358695 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a5d8f8-cf9e-4c1d-827a-d90bb14bb27e None admin] PATH_INFO: `/v3/users/swift` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:18.378123 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18a5d8f8-cf9e-4c1d-827a-d90bb14bb27e None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:18.379568 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18a5d8f8-cf9e-4c1d-827a-d90bb14bb27e None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:18.389579 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d41cebb-df9d-4f7a-9925-a179c635e265 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:18.390361 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d41cebb-df9d-4f7a-9925-a179c635e265 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:18.390735 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d41cebb-df9d-4f7a-9925-a179c635e265 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:18.391105 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d41cebb-df9d-4f7a-9925-a179c635e265 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:18.399442 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d41cebb-df9d-4f7a-9925-a179c635e265 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:18.399639 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-18a5d8f8-cf9e-4c1d-827a-d90bb14bb27e None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 25 00:42:18.399639 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:18.399639 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:18.399639 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:18.399639 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:18.399639 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:18.399639 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:18.399639 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:18.399639 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:18.399639 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:18.399639 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:18.399639 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:18.399639 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 00:42:18.399639 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 00:42:18.399639 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 00:42:18.399639 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 00:42:18.400786 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:18.400786 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:18.400786 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 00:42:18.400786 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:18.400786 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 00:42:18.400786 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:18.400786 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:18.400786 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:18.400786 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:18.400786 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:18.400786 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:18.400786 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:18.400786 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:18.400786 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:18.400786 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:18.400786 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:18.400786 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:18.400786 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:18.401963 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:18.401963 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:18.401963 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 00:42:18.401963 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 00:42:18.401963 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 00:42:18.401963 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 00:42:18.401963 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 00:42:18.401963 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 00:42:18.401963 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 25 00:42:18.401963 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:18.403018 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d41cebb-df9d-4f7a-9925-a179c635e265 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:18.409707 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 301/620] 10.222.0.52 () {56 vars in 1096 bytes} [Sun Sep 25 00:42:18 2022] GET /identity/v3/users/swift => generated 83 bytes in 284 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:42:18.415570 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-1d41cebb-df9d-4f7a-9925-a179c635e265 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 25 00:42:18.415570 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:18.415570 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:18.415570 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:18.415570 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:18.415570 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:18.415570 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:18.415570 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:18.415570 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:18.415570 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:18.415570 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:18.415570 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:18.415570 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 00:42:18.415570 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 00:42:18.415570 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 00:42:18.415570 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 00:42:18.417390 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:18.417390 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:18.417390 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 00:42:18.417390 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:18.417390 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 00:42:18.417390 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:18.417390 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:18.417390 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:18.417390 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:18.417390 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:18.417390 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:18.417390 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:18.417390 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:18.417390 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:18.417390 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:18.417390 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:18.417390 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:18.417390 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:18.419585 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:18.419585 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:18.419585 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 00:42:18.419585 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 00:42:18.419585 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 00:42:18.419585 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 00:42:18.419585 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 00:42:18.419585 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 00:42:18.419585 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 25 00:42:18.419585 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:18.420827 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a5d8f8-cf9e-4c1d-827a-d90bb14bb27e None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:18.423510 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 320/621] 10.222.0.52 () {56 vars in 1093 bytes} [Sun Sep 25 00:42:18 2022] GET /identity/v3/users/nova => generated 82 bytes in 284 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:42:18.424586 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d41cebb-df9d-4f7a-9925-a179c635e265 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:18.504712 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:18.543248 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:18.577887 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a5d8f8-cf9e-4c1d-827a-d90bb14bb27e None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:18.626759 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d41cebb-df9d-4f7a-9925-a179c635e265 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:18.630694 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b5d271-d8ba-4422-ab6b-cdf32d3e6ff4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:18.631406 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b5d271-d8ba-4422-ab6b-cdf32d3e6ff4 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:18.631781 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b5d271-d8ba-4422-ab6b-cdf32d3e6ff4 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:18.632145 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b5d271-d8ba-4422-ab6b-cdf32d3e6ff4 None admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:18.633971 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65b5d271-d8ba-4422-ab6b-cdf32d3e6ff4 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:18.634396 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65b5d271-d8ba-4422-ab6b-cdf32d3e6ff4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:18.636378 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65b5d271-d8ba-4422-ab6b-cdf32d3e6ff4 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:18.665293 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 302/622] 10.222.0.52 () {56 vars in 1135 bytes} [Sun Sep 25 00:42:18 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:18.665293 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838c0110-f4fb-44a5-9822-10f4c7dbf702 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:18.665293 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838c0110-f4fb-44a5-9822-10f4c7dbf702 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:18.665293 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838c0110-f4fb-44a5-9822-10f4c7dbf702 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:18.666962 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 303/623] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:42:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:42:18.670168 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838c0110-f4fb-44a5-9822-10f4c7dbf702 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:18.675782 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50ba6015-1fb1-48ec-a7ff-47e02447fa3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:18.676405 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ba6015-1fb1-48ec-a7ff-47e02447fa3f None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:18.676678 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ba6015-1fb1-48ec-a7ff-47e02447fa3f None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:18.676944 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ba6015-1fb1-48ec-a7ff-47e02447fa3f None admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:18.677730 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50ba6015-1fb1-48ec-a7ff-47e02447fa3f None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:18.678058 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50ba6015-1fb1-48ec-a7ff-47e02447fa3f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:18.680072 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50ba6015-1fb1-48ec-a7ff-47e02447fa3f None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:18.705345 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 321/624] 10.222.0.52 () {56 vars in 1133 bytes} [Sun Sep 25 00:42:18 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:42:18.708184 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d983db-bd3b-4b13-9287-032b7c953033 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:18.709741 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d983db-bd3b-4b13-9287-032b7c953033 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:18.710025 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d983db-bd3b-4b13-9287-032b7c953033 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:18.730121 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-74d983db-bd3b-4b13-9287-032b7c953033 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:42:18.733979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-74d983db-bd3b-4b13-9287-032b7c953033 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:42:18.809138 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838c0110-f4fb-44a5-9822-10f4c7dbf702 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:18.822569 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 322/625] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:42:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:42:18.823775 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d983db-bd3b-4b13-9287-032b7c953033 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:18.844626 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:18.865799 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:18.877575 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c30a57-6b62-474d-b68d-857aa51ef731 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:18.878632 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c30a57-6b62-474d-b68d-857aa51ef731 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:18.879113 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c30a57-6b62-474d-b68d-857aa51ef731 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:18.879676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c30a57-6b62-474d-b68d-857aa51ef731 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:18.903118 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c30a57-6b62-474d-b68d-857aa51ef731 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:18.905002 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c30a57-6b62-474d-b68d-857aa51ef731 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:18.932633 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-56c30a57-6b62-474d-b68d-857aa51ef731 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:18.932633 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:18.932633 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:18.932633 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:18.932633 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:18.932633 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:18.932633 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:18.932633 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:18.932633 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:18.932633 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:18.932633 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:18.932633 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:18.932633 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:18.932633 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:18.932633 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:18.932633 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:18.933703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:18.933703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:18.933703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:18.933703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:18.933703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:18.933703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:18.933703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:18.933703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:18.933703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:18.933703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:18.933703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:18.933703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:18.933703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:18.933703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:18.933703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:18.933703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:18.933703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:18.934634 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:18.934634 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:18.934634 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:18.941425 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d983db-bd3b-4b13-9287-032b7c953033 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:18.942736 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 304/626] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 273 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:18.956924 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c30a57-6b62-474d-b68d-857aa51ef731 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:18.982650 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c1a43c-3c4c-4943-be92-eca879ccbca3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:18.983258 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c1a43c-3c4c-4943-be92-eca879ccbca3 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:18.983534 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c1a43c-3c4c-4943-be92-eca879ccbca3 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:18.983798 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c1a43c-3c4c-4943-be92-eca879ccbca3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:19.066075 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03c1a43c-3c4c-4943-be92-eca879ccbca3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:19.067578 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03c1a43c-3c4c-4943-be92-eca879ccbca3 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:19.109632 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:19.166840 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c30a57-6b62-474d-b68d-857aa51ef731 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:19.185158 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 323/627] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:42:18 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 362 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 00:42:19.194429 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c1a43c-3c4c-4943-be92-eca879ccbca3 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:19.229785 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af0d96e-9f2f-40ef-93d9-f2385b4ec210 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:19.230753 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af0d96e-9f2f-40ef-93d9-f2385b4ec210 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:19.231297 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af0d96e-9f2f-40ef-93d9-f2385b4ec210 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:19.231797 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af0d96e-9f2f-40ef-93d9-f2385b4ec210 None admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:19.232709 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af0d96e-9f2f-40ef-93d9-f2385b4ec210 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:19.233204 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af0d96e-9f2f-40ef-93d9-f2385b4ec210 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:19.235032 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af0d96e-9f2f-40ef-93d9-f2385b4ec210 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:19.249672 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 305/628] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 293 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:19.251564 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265c5f7c-06a0-430a-a6e5-59b28c3cba40 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:19.252114 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265c5f7c-06a0-430a-a6e5-59b28c3cba40 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:19.252585 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265c5f7c-06a0-430a-a6e5-59b28c3cba40 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:19.254732 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 306/629] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:42:19 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:42:19.256727 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-265c5f7c-06a0-430a-a6e5-59b28c3cba40 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:19.293772 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:19.336899 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c1a43c-3c4c-4943-be92-eca879ccbca3 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:19.375745 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29d6f0d8-1db7-43ef-9fb2-153b0f75f027 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:19.376381 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d6f0d8-1db7-43ef-9fb2-153b0f75f027 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:19.376655 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d6f0d8-1db7-43ef-9fb2-153b0f75f027 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:19.376935 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d6f0d8-1db7-43ef-9fb2-153b0f75f027 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:19.377778 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-265c5f7c-06a0-430a-a6e5-59b28c3cba40 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:19.380092 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:19.389156 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29d6f0d8-1db7-43ef-9fb2-153b0f75f027 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:19.391404 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29d6f0d8-1db7-43ef-9fb2-153b0f75f027 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:19.407130 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-29d6f0d8-1db7-43ef-9fb2-153b0f75f027 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:19.407130 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:19.407130 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:19.407130 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:19.407130 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:19.407130 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:19.407130 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:19.407130 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:19.407130 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:19.407130 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:19.407130 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:19.407130 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:19.407130 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:19.407130 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:19.407130 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:19.407130 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:19.408140 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:19.408140 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:19.408140 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:19.408140 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:19.408140 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:19.408140 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:19.408140 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:19.408140 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:19.408140 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:19.408140 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:19.408140 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:19.408140 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:19.408140 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:19.408140 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:19.408140 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:19.408140 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:19.408140 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:19.409057 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:19.409057 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:19.409057 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:19.409057 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 324/630] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:19.409860 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2ea4cb-f76d-444c-b782-669cca6c63e2 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:19.409860 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2ea4cb-f76d-444c-b782-669cca6c63e2 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:19.410036 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2ea4cb-f76d-444c-b782-669cca6c63e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:19.416074 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ceabaf3-285d-43b0-8e87-27aac8433eb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:19.416709 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ceabaf3-285d-43b0-8e87-27aac8433eb3 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:19.416983 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ceabaf3-285d-43b0-8e87-27aac8433eb3 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:19.417245 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ceabaf3-285d-43b0-8e87-27aac8433eb3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:19.424450 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ceabaf3-285d-43b0-8e87-27aac8433eb3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:19.425957 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ceabaf3-285d-43b0-8e87-27aac8433eb3 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:19.429970 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-3c2ea4cb-f76d-444c-b782-669cca6c63e2 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:42:19.431400 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-3c2ea4cb-f76d-444c-b782-669cca6c63e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:42:19.439303 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-5ceabaf3-285d-43b0-8e87-27aac8433eb3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:42:19.439303 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:19.439303 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:19.439303 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:19.439303 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:19.439303 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:19.439303 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:19.439303 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:19.439303 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:19.439303 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:19.439303 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:19.439303 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:19.439303 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 00:42:19.439303 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 00:42:19.439303 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 00:42:19.440556 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 00:42:19.440556 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:19.440556 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:19.440556 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:19.440556 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:19.440556 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:19.440556 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:19.440556 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:19.440556 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:19.440556 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:19.440556 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:19.440556 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:19.440556 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:19.440556 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:19.440556 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:19.440556 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:19.440556 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:19.441691 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 00:42:19.441691 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 00:42:19.441691 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 00:42:19.441691 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 00:42:19.441691 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 00:42:19.441691 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 00:42:19.441691 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:42:19.441691 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:19.441691 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 307/631] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:42:19 2022] GET /identity/v3/projects/service => generated 88 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:42:19.441691 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ceabaf3-285d-43b0-8e87-27aac8433eb3 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:19.505687 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 325/632] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:42:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:42:19.506947 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba04be1-8bc4-4393-8c99-d9f1925a8a81 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:19.507766 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba04be1-8bc4-4393-8c99-d9f1925a8a81 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:19.508041 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba04be1-8bc4-4393-8c99-d9f1925a8a81 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:19.509292 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 326/633] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:42:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:42:19.513537 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba04be1-8bc4-4393-8c99-d9f1925a8a81 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:19.547314 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:19.558374 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ceabaf3-285d-43b0-8e87-27aac8433eb3 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:19.594449 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76009abd-3c5c-4a40-bec3-cfa47dae7063 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:19.595750 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76009abd-3c5c-4a40-bec3-cfa47dae7063 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:19.596031 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76009abd-3c5c-4a40-bec3-cfa47dae7063 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:19.596302 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76009abd-3c5c-4a40-bec3-cfa47dae7063 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:19.607573 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76009abd-3c5c-4a40-bec3-cfa47dae7063 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:42:19.612896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-76009abd-3c5c-4a40-bec3-cfa47dae7063 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 25 00:42:19.612896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:19.612896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:19.612896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:19.612896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:19.612896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:19.612896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:19.612896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:19.612896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:19.612896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:19.612896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:19.612896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:19.612896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:42:19.612896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:42:19.612896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:42:19.612896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:42:19.614152 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:42:19.614152 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:42:19.614152 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:42:19.614152 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:42:19.614152 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:19.614152 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:19.614152 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:19.614152 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:19.614152 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:19.614152 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:19.614152 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:19.614152 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:19.614152 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:19.614152 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:19.614152 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:19.614152 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:19.615013 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:19.615013 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:19.615013 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:19.615013 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:19.615013 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:42:19.615013 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:42:19.615013 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:42:19.615013 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:42:19.615013 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:42:19.615013 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:42:19.615013 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 25 00:42:19.615013 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:19.615013 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 308/634] 10.222.0.52 () {56 vars in 1102 bytes} [Sun Sep 25 00:42:19 2022] GET /identity/v3/roles/service => generated 85 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:19.616326 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76009abd-3c5c-4a40-bec3-cfa47dae7063 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:19.637124 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba04be1-8bc4-4393-8c99-d9f1925a8a81 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:19.655877 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:19.675129 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79665e2e-d236-414f-a1c2-972b8af35d2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:19.675734 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79665e2e-d236-414f-a1c2-972b8af35d2b None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:19.676011 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79665e2e-d236-414f-a1c2-972b8af35d2b None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:19.676453 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79665e2e-d236-414f-a1c2-972b8af35d2b None admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:19.677589 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79665e2e-d236-414f-a1c2-972b8af35d2b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:19.678144 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79665e2e-d236-414f-a1c2-972b8af35d2b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:19.680408 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79665e2e-d236-414f-a1c2-972b8af35d2b None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:19.693329 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 327/635] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:19.694557 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2a8c23-0159-45af-85af-4c80e0c49c0e None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:19.695032 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2a8c23-0159-45af-85af-4c80e0c49c0e None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:19.695334 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2a8c23-0159-45af-85af-4c80e0c49c0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:19.715856 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-df2a8c23-0159-45af-85af-4c80e0c49c0e None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:42:19.717058 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-df2a8c23-0159-45af-85af-4c80e0c49c0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:42:19.724721 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76009abd-3c5c-4a40-bec3-cfa47dae7063 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:19.762893 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f773325-3c23-4177-b1f1-e2f82ae72c6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:19.763506 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f773325-3c23-4177-b1f1-e2f82ae72c6d None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:19.764098 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f773325-3c23-4177-b1f1-e2f82ae72c6d None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:19.764384 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f773325-3c23-4177-b1f1-e2f82ae72c6d None admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:19.765075 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f773325-3c23-4177-b1f1-e2f82ae72c6d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:19.765422 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f773325-3c23-4177-b1f1-e2f82ae72c6d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:19.766851 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f773325-3c23-4177-b1f1-e2f82ae72c6d None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:19.785572 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 309/636] 10.222.0.52 () {56 vars in 1148 bytes} [Sun Sep 25 00:42:19 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:19.786389 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f773325-3c23-4177-b1f1-e2f82ae72c6d None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:19.794034 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 328/637] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:42:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:42:19.794912 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2a8c23-0159-45af-85af-4c80e0c49c0e None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:19.802498 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:19.896236 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f773325-3c23-4177-b1f1-e2f82ae72c6d None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:19.903888 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2a8c23-0159-45af-85af-4c80e0c49c0e None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:19.915593 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:19.940972 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b932d3a1-af41-4c47-9251-322a4950a0f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:19.941853 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b932d3a1-af41-4c47-9251-322a4950a0f0 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:19.942131 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b932d3a1-af41-4c47-9251-322a4950a0f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:19.942401 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b932d3a1-af41-4c47-9251-322a4950a0f0 None admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:19.943065 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b932d3a1-af41-4c47-9251-322a4950a0f0 None admin] RBAC: Adding query filter params (name=service) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:19.943368 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b932d3a1-af41-4c47-9251-322a4950a0f0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:19.944204 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2a014fa-3970-4b66-9e78-4c57d1d50a5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:19.945466 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b932d3a1-af41-4c47-9251-322a4950a0f0 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:19.945597 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a014fa-3970-4b66-9e78-4c57d1d50a5c None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:19.945932 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a014fa-3970-4b66-9e78-4c57d1d50a5c None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:19.946136 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a014fa-3970-4b66-9e78-4c57d1d50a5c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:19.958055 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 310/638] 10.222.0.52 () {56 vars in 1103 bytes} [Sun Sep 25 00:42:19 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:19.958252 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2a014fa-3970-4b66-9e78-4c57d1d50a5c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:19.958347 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b932d3a1-af41-4c47-9251-322a4950a0f0 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:19.958446 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2a014fa-3970-4b66-9e78-4c57d1d50a5c None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:19.970801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-e2a014fa-3970-4b66-9e78-4c57d1d50a5c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:42:19.970801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:19.970801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:19.970801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:19.970801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:19.970801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:19.970801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:19.970801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:19.970801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:19.970801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:19.970801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:19.970801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:19.970801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 00:42:19.970801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 00:42:19.970801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 00:42:19.971908 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 00:42:19.971908 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:19.971908 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:19.971908 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:19.971908 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:19.971908 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:19.971908 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:19.971908 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:19.971908 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:19.971908 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:19.971908 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:19.971908 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:19.971908 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:19.971908 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:19.971908 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:19.971908 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:19.971908 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:19.972820 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 00:42:19.972820 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 00:42:19.972820 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 00:42:19.972820 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 00:42:19.972820 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 00:42:19.972820 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 00:42:19.972820 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:42:19.972820 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:19.972820 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 329/639] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:42:19 2022] GET /identity/v3/projects/service => generated 88 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:19.973332 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2a014fa-3970-4b66-9e78-4c57d1d50a5c None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:20.051861 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:20.077776 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b932d3a1-af41-4c47-9251-322a4950a0f0 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:20.078125 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2a014fa-3970-4b66-9e78-4c57d1d50a5c None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:20.111630 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af291d1e-0c47-4c3a-a8eb-af65eab07f6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:20.111905 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbff2160-a8bd-40d8-98d1-39c04e20442a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:20.112259 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af291d1e-0c47-4c3a-a8eb-af65eab07f6b None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:20.112537 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af291d1e-0c47-4c3a-a8eb-af65eab07f6b None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:20.112631 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbff2160-a8bd-40d8-98d1-39c04e20442a None admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:20.112807 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbff2160-a8bd-40d8-98d1-39c04e20442a None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:20.112905 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af291d1e-0c47-4c3a-a8eb-af65eab07f6b None admin] PATH_INFO: `/v3/roles/service` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:20.113352 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbff2160-a8bd-40d8-98d1-39c04e20442a None admin] PATH_INFO: `/v3/projects/9db0fd07fc494203b2ada13f3cd046d3/users/da606b3906dd4171bd76fd281e03e258/roles/a8e34e460e8a4ed7bcaa54da3c2d5a0c` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:20.123959 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af291d1e-0c47-4c3a-a8eb-af65eab07f6b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:42:20.128544 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbff2160-a8bd-40d8-98d1-39c04e20442a None admin] RBAC: Authorizing `identity:create_grant(project_id=9db0fd07fc494203b2ada13f3cd046d3, user_id=da606b3906dd4171bd76fd281e03e258, role_id=a8e34e460e8a4ed7bcaa54da3c2d5a0c)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:20.129273 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-af291d1e-0c47-4c3a-a8eb-af65eab07f6b None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 25 00:42:20.129273 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:20.129273 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:20.129273 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:20.129273 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:20.129273 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:20.129273 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:20.129273 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:20.129273 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:20.129273 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:20.129273 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:20.129273 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:20.129273 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:42:20.129273 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:42:20.129273 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:42:20.129273 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:42:20.130359 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:42:20.130359 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:42:20.130359 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:42:20.130359 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:42:20.130359 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:20.130359 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:20.130359 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:20.130359 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:20.130359 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:20.130359 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:20.130359 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:20.130359 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:20.130359 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:20.130359 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:20.130359 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:20.130359 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:20.134286 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:20.134286 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:20.134286 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:20.134286 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:20.134286 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:42:20.134286 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:42:20.134286 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:42:20.134286 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:42:20.134286 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:42:20.134286 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:42:20.134286 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 25 00:42:20.134286 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:20.134948 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbff2160-a8bd-40d8-98d1-39c04e20442a None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:20.135023 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 311/640] 10.222.0.52 () {56 vars in 1102 bytes} [Sun Sep 25 00:42:19 2022] GET /identity/v3/roles/service => generated 85 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:20.135023 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af291d1e-0c47-4c3a-a8eb-af65eab07f6b None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:20.145999 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 330/641] 10.222.0.52 () {58 vars in 1439 bytes} [Sun Sep 25 00:42:19 2022] PUT /identity/v3/projects/9db0fd07fc494203b2ada13f3cd046d3/users/da606b3906dd4171bd76fd281e03e258/roles/a8e34e460e8a4ed7bcaa54da3c2d5a0c => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:42:20.147150 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbff2160-a8bd-40d8-98d1-39c04e20442a None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:20.169534 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:20.267016 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af291d1e-0c47-4c3a-a8eb-af65eab07f6b None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:20.276128 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbff2160-a8bd-40d8-98d1-39c04e20442a None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:20.310849 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4caefce0-9605-49a6-8f72-b76b80466c1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:20.311950 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4caefce0-9605-49a6-8f72-b76b80466c1c None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:20.312447 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4caefce0-9605-49a6-8f72-b76b80466c1c None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:20.312917 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4caefce0-9605-49a6-8f72-b76b80466c1c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:20.316235 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:20.317598 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b7432b6-4138-4e5c-91fe-8a535dbecf00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:20.318360 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7432b6-4138-4e5c-91fe-8a535dbecf00 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:20.318942 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7432b6-4138-4e5c-91fe-8a535dbecf00 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:20.319329 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7432b6-4138-4e5c-91fe-8a535dbecf00 None admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:20.321461 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b7432b6-4138-4e5c-91fe-8a535dbecf00 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:20.321781 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b7432b6-4138-4e5c-91fe-8a535dbecf00 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:20.324195 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b7432b6-4138-4e5c-91fe-8a535dbecf00 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:20.330964 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4caefce0-9605-49a6-8f72-b76b80466c1c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:20.332412 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4caefce0-9605-49a6-8f72-b76b80466c1c None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:20.343133 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 331/642] 10.222.0.52 () {56 vars in 1148 bytes} [Sun Sep 25 00:42:20 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:20.343987 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b7432b6-4138-4e5c-91fe-8a535dbecf00 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:20.349723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-4caefce0-9605-49a6-8f72-b76b80466c1c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:20.349723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:20.349723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:20.349723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:20.349723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:20.349723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:20.349723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:20.349723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:20.349723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:20.349723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:20.349723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:20.349723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:20.349723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:20.349723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:20.349723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:20.349723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:20.350777 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:20.350777 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:20.350777 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:20.350777 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:20.350777 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:20.350777 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:20.350777 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:20.350777 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:20.350777 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:20.350777 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:20.350777 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:20.350777 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:20.350777 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:20.350777 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:20.350777 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:20.350777 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:20.350777 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:20.352114 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:20.352114 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:20.352114 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:20.352114 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 312/643] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:20.352114 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4caefce0-9605-49a6-8f72-b76b80466c1c None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:20.430585 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:20.456593 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b7432b6-4138-4e5c-91fe-8a535dbecf00 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:20.469987 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4caefce0-9605-49a6-8f72-b76b80466c1c None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:20.499827 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-924a5e2a-a268-4bf6-80c8-3b82008de4a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:20.500460 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-924a5e2a-a268-4bf6-80c8-3b82008de4a3 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:20.500732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-924a5e2a-a268-4bf6-80c8-3b82008de4a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:20.500995 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-924a5e2a-a268-4bf6-80c8-3b82008de4a3 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:20.512771 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-924a5e2a-a268-4bf6-80c8-3b82008de4a3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:42:20.513404 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a5a0a8-bced-4f14-9d41-427ea0d7d8c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:20.514007 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a5a0a8-bced-4f14-9d41-427ea0d7d8c4 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:20.514283 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a5a0a8-bced-4f14-9d41-427ea0d7d8c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:20.514550 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a5a0a8-bced-4f14-9d41-427ea0d7d8c4 None admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:20.515210 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a5a0a8-bced-4f14-9d41-427ea0d7d8c4 None admin] RBAC: Adding query filter params (name=service) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:20.515509 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a5a0a8-bced-4f14-9d41-427ea0d7d8c4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:20.516988 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a5a0a8-bced-4f14-9d41-427ea0d7d8c4 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:20.519586 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-924a5e2a-a268-4bf6-80c8-3b82008de4a3 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 25 00:42:20.519586 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:20.519586 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:20.519586 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:20.519586 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:20.519586 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:20.519586 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:20.519586 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:20.519586 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:20.519586 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:20.519586 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:20.519586 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:20.519586 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:42:20.519586 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:42:20.519586 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:42:20.519586 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:42:20.520489 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:42:20.520489 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:42:20.520489 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:42:20.520489 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:42:20.520489 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:20.520489 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:20.520489 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:20.520489 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:20.520489 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:20.520489 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:20.520489 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:20.520489 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:20.520489 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:20.520489 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:20.520489 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:20.520489 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:20.521516 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:20.521516 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:20.521516 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:20.521516 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:20.521516 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:42:20.521516 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:42:20.521516 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:42:20.521516 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:42:20.521516 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:42:20.521516 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:42:20.521516 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 25 00:42:20.521516 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:20.521516 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 332/644] 10.222.0.52 () {56 vars in 1102 bytes} [Sun Sep 25 00:42:20 2022] GET /identity/v3/roles/service => generated 85 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:20.523404 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-924a5e2a-a268-4bf6-80c8-3b82008de4a3 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:20.535433 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 313/645] 10.222.0.52 () {56 vars in 1103 bytes} [Sun Sep 25 00:42:20 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:20.535433 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a5a0a8-bced-4f14-9d41-427ea0d7d8c4 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:20.581914 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:20.635124 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a5a0a8-bced-4f14-9d41-427ea0d7d8c4 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:20.642660 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-924a5e2a-a268-4bf6-80c8-3b82008de4a3 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:20.676297 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da71fd68-ee9a-43a6-a08e-27ee95ec5364 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:20.676907 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da71fd68-ee9a-43a6-a08e-27ee95ec5364 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:20.677237 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da71fd68-ee9a-43a6-a08e-27ee95ec5364 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:20.677533 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da71fd68-ee9a-43a6-a08e-27ee95ec5364 None admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:20.678169 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da71fd68-ee9a-43a6-a08e-27ee95ec5364 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:20.678469 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da71fd68-ee9a-43a6-a08e-27ee95ec5364 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:20.678624 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f18e2e-84f0-4ab9-9194-c456cf90a443 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:20.679345 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f18e2e-84f0-4ab9-9194-c456cf90a443 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:20.679751 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f18e2e-84f0-4ab9-9194-c456cf90a443 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:20.680079 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da71fd68-ee9a-43a6-a08e-27ee95ec5364 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:20.680250 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f18e2e-84f0-4ab9-9194-c456cf90a443 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:20.680997 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60f18e2e-84f0-4ab9-9194-c456cf90a443 None admin] RBAC: Adding query filter params (role.id=a8e34e460e8a4ed7bcaa54da3c2d5a0c, scope.project.id=9db0fd07fc494203b2ada13f3cd046d3, user.id=f142988a4dd14a5f96eacea45c046f85) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:20.681565 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60f18e2e-84f0-4ab9-9194-c456cf90a443 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:20.682677 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60f18e2e-84f0-4ab9-9194-c456cf90a443 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:20.698082 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 333/646] 10.222.0.52 () {56 vars in 1374 bytes} [Sun Sep 25 00:42:20 2022] GET /identity/v3/role_assignments?user.id=f142988a4dd14a5f96eacea45c046f85&scope.project.id=9db0fd07fc494203b2ada13f3cd046d3&role.id=a8e34e460e8a4ed7bcaa54da3c2d5a0c => generated 608 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:20.698082 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f18e2e-84f0-4ab9-9194-c456cf90a443 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:20.698082 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:20.710298 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 314/647] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:20.711718 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da71fd68-ee9a-43a6-a08e-27ee95ec5364 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:20.820461 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f18e2e-84f0-4ab9-9194-c456cf90a443 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:20.853003 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da71fd68-ee9a-43a6-a08e-27ee95ec5364 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:20.868298 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:20.879925 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7888d442-9586-4540-bcb9-dfb06f72fdf8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:20.880553 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7888d442-9586-4540-bcb9-dfb06f72fdf8 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:20.880827 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7888d442-9586-4540-bcb9-dfb06f72fdf8 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:20.881093 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7888d442-9586-4540-bcb9-dfb06f72fdf8 None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:20.881847 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7888d442-9586-4540-bcb9-dfb06f72fdf8 None admin] RBAC: Adding query filter params (name=service) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:20.882752 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7888d442-9586-4540-bcb9-dfb06f72fdf8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:20.885366 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7888d442-9586-4540-bcb9-dfb06f72fdf8 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:20.896641 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 334/648] 10.222.0.52 () {56 vars in 1103 bytes} [Sun Sep 25 00:42:20 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:20.898123 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7888d442-9586-4540-bcb9-dfb06f72fdf8 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:20.903425 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a35d678c-c6f2-42d4-85d5-ffd8264c3a1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:20.904332 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35d678c-c6f2-42d4-85d5-ffd8264c3a1f None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:20.904608 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35d678c-c6f2-42d4-85d5-ffd8264c3a1f None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:20.904875 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35d678c-c6f2-42d4-85d5-ffd8264c3a1f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:20.919347 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a35d678c-c6f2-42d4-85d5-ffd8264c3a1f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:20.920803 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a35d678c-c6f2-42d4-85d5-ffd8264c3a1f None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:20.939943 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-a35d678c-c6f2-42d4-85d5-ffd8264c3a1f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:20.939943 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:20.939943 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:20.939943 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:20.939943 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:20.939943 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:20.939943 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:20.939943 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:20.939943 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:20.939943 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:20.939943 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:20.939943 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:20.939943 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:20.939943 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:20.939943 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:20.939943 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:20.940948 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:20.940948 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:20.940948 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:20.940948 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:20.940948 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:20.940948 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:20.940948 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:20.940948 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:20.940948 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:20.940948 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:20.940948 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:20.940948 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:20.940948 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:20.940948 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:20.940948 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:20.940948 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:20.940948 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:20.941862 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:20.941862 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:20.941862 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:20.941862 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 315/649] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:20.941862 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a35d678c-c6f2-42d4-85d5-ffd8264c3a1f None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:20.980904 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:21.045494 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7888d442-9586-4540-bcb9-dfb06f72fdf8 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:21.082292 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a35d678c-c6f2-42d4-85d5-ffd8264c3a1f None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:21.088799 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec012898-880e-43e2-afef-d98bb1bb722b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:21.089478 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec012898-880e-43e2-afef-d98bb1bb722b None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:21.089751 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec012898-880e-43e2-afef-d98bb1bb722b None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:21.090025 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec012898-880e-43e2-afef-d98bb1bb722b None admin] PATH_INFO: `/v3/users/glance` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:21.098389 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec012898-880e-43e2-afef-d98bb1bb722b None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:21.099876 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec012898-880e-43e2-afef-d98bb1bb722b None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:21.111829 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-ec012898-880e-43e2-afef-d98bb1bb722b None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 25 00:42:21.111829 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:21.111829 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:21.111829 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:21.111829 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:21.111829 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:21.111829 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:21.111829 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:21.111829 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:21.111829 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:21.111829 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:21.111829 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:21.111829 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 00:42:21.111829 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 00:42:21.111829 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 00:42:21.111829 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 00:42:21.113005 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:21.113005 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:21.113005 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 00:42:21.113005 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:21.113005 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 00:42:21.113005 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:21.113005 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:21.113005 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:21.113005 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:21.113005 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:21.113005 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:21.113005 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:21.113005 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:21.113005 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:21.113005 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:21.113005 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:21.113005 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:21.113005 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:21.114419 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:21.114419 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:21.114419 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 00:42:21.114419 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 00:42:21.114419 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 00:42:21.114419 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 00:42:21.114419 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 00:42:21.114419 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 00:42:21.114419 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 25 00:42:21.114419 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:21.114419 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 335/650] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:42:20 2022] GET /identity/v3/users/glance => generated 84 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:21.114419 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec012898-880e-43e2-afef-d98bb1bb722b None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:21.131662 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a99021-e633-4e61-9620-2336f747ba73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:21.132634 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a99021-e633-4e61-9620-2336f747ba73 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:21.133498 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a99021-e633-4e61-9620-2336f747ba73 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:21.133938 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a99021-e633-4e61-9620-2336f747ba73 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:21.152403 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a99021-e633-4e61-9620-2336f747ba73 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:21.154788 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a99021-e633-4e61-9620-2336f747ba73 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:21.178535 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-59a99021-e633-4e61-9620-2336f747ba73 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:21.178535 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:21.178535 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:21.178535 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:21.178535 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:21.178535 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:21.178535 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:21.178535 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:21.178535 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:21.178535 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:21.178535 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:21.178535 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:21.178535 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:21.178535 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:21.178535 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:21.178535 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:21.180263 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:21.180263 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:21.180263 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:21.180263 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:21.180263 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:21.180263 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:21.180263 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:21.180263 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:21.180263 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:21.180263 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:21.180263 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:21.180263 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:21.180263 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:21.180263 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:21.180263 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:21.180263 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:21.180263 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:21.181400 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:21.181400 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:21.181400 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:21.183906 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 316/651] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 242 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:21.184991 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a99021-e633-4e61-9620-2336f747ba73 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:21.204014 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:21.252340 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec012898-880e-43e2-afef-d98bb1bb722b None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:21.280018 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:21.293372 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09af0807-9d7b-4df9-943f-1728e50b96a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:21.294471 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09af0807-9d7b-4df9-943f-1728e50b96a0 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:21.294742 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09af0807-9d7b-4df9-943f-1728e50b96a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:21.295007 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09af0807-9d7b-4df9-943f-1728e50b96a0 None admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:21.296030 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09af0807-9d7b-4df9-943f-1728e50b96a0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:21.296330 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09af0807-9d7b-4df9-943f-1728e50b96a0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:21.297793 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09af0807-9d7b-4df9-943f-1728e50b96a0 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:21.305712 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a99021-e633-4e61-9620-2336f747ba73 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:21.317973 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 336/652] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:21.317973 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09af0807-9d7b-4df9-943f-1728e50b96a0 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:21.352496 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c3359c1-f8d4-46c6-a05c-c95115a4fed7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:21.353442 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3359c1-f8d4-46c6-a05c-c95115a4fed7 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:21.353926 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3359c1-f8d4-46c6-a05c-c95115a4fed7 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:21.354343 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3359c1-f8d4-46c6-a05c-c95115a4fed7 None admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:21.355287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c3359c1-f8d4-46c6-a05c-c95115a4fed7 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:21.355736 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c3359c1-f8d4-46c6-a05c-c95115a4fed7 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:21.357383 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c3359c1-f8d4-46c6-a05c-c95115a4fed7 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:21.379060 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 317/653] 10.222.0.52 () {56 vars in 1137 bytes} [Sun Sep 25 00:42:21 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:21.380298 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c3359c1-f8d4-46c6-a05c-c95115a4fed7 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:21.445771 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09af0807-9d7b-4df9-943f-1728e50b96a0 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:21.458697 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:21.490763 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e4c627-7f0b-403d-a542-d85c582f1295 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:21.491878 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e4c627-7f0b-403d-a542-d85c582f1295 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:21.492234 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e4c627-7f0b-403d-a542-d85c582f1295 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:21.492500 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e4c627-7f0b-403d-a542-d85c582f1295 None admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:21.493713 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45e4c627-7f0b-403d-a542-d85c582f1295 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:21.493937 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45e4c627-7f0b-403d-a542-d85c582f1295 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:21.495688 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45e4c627-7f0b-403d-a542-d85c582f1295 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:21.503054 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c3359c1-f8d4-46c6-a05c-c95115a4fed7 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:21.510331 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 337/654] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:21.511013 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e4c627-7f0b-403d-a542-d85c582f1295 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:21.545818 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb0bf01-c911-4c46-bb7a-add1270a6871 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:21.546788 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb0bf01-c911-4c46-bb7a-add1270a6871 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:21.547245 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb0bf01-c911-4c46-bb7a-add1270a6871 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:21.547738 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb0bf01-c911-4c46-bb7a-add1270a6871 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:21.555876 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdb0bf01-c911-4c46-bb7a-add1270a6871 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:21.557678 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdb0bf01-c911-4c46-bb7a-add1270a6871 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:21.571817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-cdb0bf01-c911-4c46-bb7a-add1270a6871 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 25 00:42:21.571817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:21.571817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:21.571817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:21.571817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:21.571817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:21.571817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:21.571817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:21.571817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:21.571817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:21.571817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:21.571817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:21.571817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 00:42:21.571817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 00:42:21.571817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 00:42:21.571817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 00:42:21.572838 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:21.572838 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:21.572838 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 00:42:21.572838 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:21.572838 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 00:42:21.572838 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:21.572838 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:21.572838 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:21.572838 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:21.572838 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:21.572838 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:21.572838 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:21.572838 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:21.572838 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:21.572838 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:21.572838 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:21.572838 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:21.572838 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:21.573834 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:21.573834 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:21.573834 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 00:42:21.573834 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 00:42:21.573834 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 00:42:21.573834 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 00:42:21.573834 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 00:42:21.573834 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 00:42:21.573834 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 25 00:42:21.573834 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:21.580438 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 318/655] 10.222.0.52 () {56 vars in 1102 bytes} [Sun Sep 25 00:42:21 2022] GET /identity/v3/users/neutron => generated 85 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:21.581675 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb0bf01-c911-4c46-bb7a-add1270a6871 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:21.589696 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:21.641102 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e4c627-7f0b-403d-a542-d85c582f1295 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:21.684106 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-267d01ab-d64f-45c3-8716-e677af87347a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:21.685103 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267d01ab-d64f-45c3-8716-e677af87347a None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:21.685103 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267d01ab-d64f-45c3-8716-e677af87347a None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:21.685395 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267d01ab-d64f-45c3-8716-e677af87347a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:21.703032 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb0bf01-c911-4c46-bb7a-add1270a6871 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:21.737135 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:21.745431 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc494ea-1cc0-472e-83d6-21fd5fcc6fa4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:21.746292 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc494ea-1cc0-472e-83d6-21fd5fcc6fa4 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:21.746720 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc494ea-1cc0-472e-83d6-21fd5fcc6fa4 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:21.747170 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc494ea-1cc0-472e-83d6-21fd5fcc6fa4 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:21.755856 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bc494ea-1cc0-472e-83d6-21fd5fcc6fa4 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:21.758214 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bc494ea-1cc0-472e-83d6-21fd5fcc6fa4 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:21.766327 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-267d01ab-d64f-45c3-8716-e677af87347a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:21.767838 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-267d01ab-d64f-45c3-8716-e677af87347a None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:21.771565 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-2bc494ea-1cc0-472e-83d6-21fd5fcc6fa4 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 25 00:42:21.771565 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:21.771565 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:21.771565 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:21.771565 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:21.771565 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:21.771565 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:21.771565 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:21.771565 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:21.771565 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:21.771565 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:21.771565 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:21.771565 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 00:42:21.771565 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 00:42:21.771565 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 00:42:21.771565 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 00:42:21.772818 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:21.772818 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:21.772818 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 00:42:21.772818 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:21.772818 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 00:42:21.772818 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:21.772818 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:21.772818 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:21.772818 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:21.772818 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:21.772818 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:21.772818 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:21.772818 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:21.772818 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:21.772818 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:21.772818 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:21.772818 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:21.772818 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:21.774315 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:21.774315 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:21.774315 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 00:42:21.774315 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 00:42:21.774315 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 00:42:21.774315 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 00:42:21.774315 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 00:42:21.774315 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 00:42:21.774315 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 25 00:42:21.774315 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:21.776499 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 319/656] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:42:21 2022] GET /identity/v3/users/cinder => generated 84 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:21.777606 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc494ea-1cc0-472e-83d6-21fd5fcc6fa4 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:21.829661 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:21.898778 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 338/657] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:42:21 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 388 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 00:42:21.900441 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-267d01ab-d64f-45c3-8716-e677af87347a None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:21.907244 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc494ea-1cc0-472e-83d6-21fd5fcc6fa4 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:21.951444 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df628969-2902-460c-84da-536a55385a1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:21.952535 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df628969-2902-460c-84da-536a55385a1c None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:21.953040 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df628969-2902-460c-84da-536a55385a1c None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:21.953545 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df628969-2902-460c-84da-536a55385a1c None admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:21.954616 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df628969-2902-460c-84da-536a55385a1c None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:21.955252 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df628969-2902-460c-84da-536a55385a1c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:21.956904 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df628969-2902-460c-84da-536a55385a1c None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:21.989158 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 320/658] 10.222.0.52 () {56 vars in 1139 bytes} [Sun Sep 25 00:42:21 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:21.990437 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df628969-2902-460c-84da-536a55385a1c None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:22.035097 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-267d01ab-d64f-45c3-8716-e677af87347a None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:22.056079 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:22.080500 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-584e8df2-412d-40d7-b656-2d8e5c592c5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:22.081109 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584e8df2-412d-40d7-b656-2d8e5c592c5a None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:22.081432 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584e8df2-412d-40d7-b656-2d8e5c592c5a None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:22.081703 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584e8df2-412d-40d7-b656-2d8e5c592c5a None admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:22.082781 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-584e8df2-412d-40d7-b656-2d8e5c592c5a None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:22.082891 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-584e8df2-412d-40d7-b656-2d8e5c592c5a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:22.084488 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-584e8df2-412d-40d7-b656-2d8e5c592c5a None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:22.116420 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 339/659] 10.222.0.52 () {56 vars in 1137 bytes} [Sun Sep 25 00:42:21 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:22.119378 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-584e8df2-412d-40d7-b656-2d8e5c592c5a None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:22.122786 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:22.130395 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df628969-2902-460c-84da-536a55385a1c None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:22.170609 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1c3cb9c-013a-4e7c-8523-857eeb253d9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:22.171751 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c3cb9c-013a-4e7c-8523-857eeb253d9d None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:22.171751 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c3cb9c-013a-4e7c-8523-857eeb253d9d None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:22.171924 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c3cb9c-013a-4e7c-8523-857eeb253d9d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:22.184093 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1c3cb9c-013a-4e7c-8523-857eeb253d9d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:22.185571 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1c3cb9c-013a-4e7c-8523-857eeb253d9d None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:22.203472 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-a1c3cb9c-013a-4e7c-8523-857eeb253d9d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:22.203472 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:22.203472 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:22.203472 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:22.203472 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:22.203472 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:22.203472 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:22.203472 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:22.203472 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:22.203472 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:22.203472 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:22.203472 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:22.203472 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:22.203472 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:22.203472 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:22.203472 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:22.204516 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:22.204516 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:22.204516 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:22.204516 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:22.204516 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:22.204516 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:22.204516 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:22.204516 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:22.204516 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:22.204516 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:22.204516 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:22.204516 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:22.204516 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:22.204516 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:22.204516 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:22.204516 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:22.204516 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:22.205425 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:22.205425 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:22.205425 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:22.209474 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 321/660] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:22.210826 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1c3cb9c-013a-4e7c-8523-857eeb253d9d None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:22.251297 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-584e8df2-412d-40d7-b656-2d8e5c592c5a None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:22.298760 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69f6c926-8b63-4f3e-8bd7-7b2a63c625a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:22.299813 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f6c926-8b63-4f3e-8bd7-7b2a63c625a1 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:22.300293 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f6c926-8b63-4f3e-8bd7-7b2a63c625a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:22.300761 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f6c926-8b63-4f3e-8bd7-7b2a63c625a1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:22.331291 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:22.336081 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1c3cb9c-013a-4e7c-8523-857eeb253d9d None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:22.382836 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff3d136-a548-4164-9dd5-8246864d9c4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:22.383606 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff3d136-a548-4164-9dd5-8246864d9c4d None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:22.383942 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff3d136-a548-4164-9dd5-8246864d9c4d None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:22.384264 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff3d136-a548-4164-9dd5-8246864d9c4d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:22.397410 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69f6c926-8b63-4f3e-8bd7-7b2a63c625a1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:22.400477 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69f6c926-8b63-4f3e-8bd7-7b2a63c625a1 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:22.403289 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:22.466081 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bff3d136-a548-4164-9dd5-8246864d9c4d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:22.467582 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bff3d136-a548-4164-9dd5-8246864d9c4d None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:22.544800 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 340/661] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:42:22 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 426 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 00:42:22.545949 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69f6c926-8b63-4f3e-8bd7-7b2a63c625a1 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:22.591318 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:22.602376 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 322/662] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:42:22 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 392 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 00:42:22.604492 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8637a07f-3376-4ee3-befc-d94bf4cf375d None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:22.604771 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8637a07f-3376-4ee3-befc-d94bf4cf375d None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:22.605068 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8637a07f-3376-4ee3-befc-d94bf4cf375d None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:22.609392 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 323/663] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:42:22 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:42:22.610438 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8637a07f-3376-4ee3-befc-d94bf4cf375d None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:22.669231 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69f6c926-8b63-4f3e-8bd7-7b2a63c625a1 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:22.671031 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:22.709607 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2fac2cc-50c3-42b4-951c-640c0c651f44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:22.711015 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2fac2cc-50c3-42b4-951c-640c0c651f44 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:22.711627 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2fac2cc-50c3-42b4-951c-640c0c651f44 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:22.712086 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2fac2cc-50c3-42b4-951c-640c0c651f44 None admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:22.712910 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2fac2cc-50c3-42b4-951c-640c0c651f44 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:22.713417 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2fac2cc-50c3-42b4-951c-640c0c651f44 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:22.714951 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2fac2cc-50c3-42b4-951c-640c0c651f44 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:22.731009 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 341/664] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:22.732109 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2fac2cc-50c3-42b4-951c-640c0c651f44 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:22.737101 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8637a07f-3376-4ee3-befc-d94bf4cf375d None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:22.783443 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2edcabc-5be2-4445-8708-d0ad89f9d6d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:22.784074 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2edcabc-5be2-4445-8708-d0ad89f9d6d1 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:22.784345 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2edcabc-5be2-4445-8708-d0ad89f9d6d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:22.784613 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2edcabc-5be2-4445-8708-d0ad89f9d6d1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:22.797999 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2edcabc-5be2-4445-8708-d0ad89f9d6d1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:22.799627 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2edcabc-5be2-4445-8708-d0ad89f9d6d1 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:22.816287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-f2edcabc-5be2-4445-8708-d0ad89f9d6d1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:22.816287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:22.816287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:22.816287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:22.816287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:22.816287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:22.816287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:22.816287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:22.816287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:22.816287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:22.816287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:22.816287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:22.816287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:22.816287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:22.816287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:22.816287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:22.817519 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:22.817519 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:22.817519 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:22.817519 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:22.817519 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:22.817519 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:22.817519 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:22.817519 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:22.817519 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:22.817519 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:22.817519 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:22.817519 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:22.817519 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:22.817519 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:22.817519 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:22.817519 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:22.817519 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:22.819043 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:22.819043 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:22.819043 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:22.819043 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 324/665] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:22.819043 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e23c7a-ef08-44b8-8a86-f30abc234ff4 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:22.819043 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e23c7a-ef08-44b8-8a86-f30abc234ff4 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:22.819388 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e23c7a-ef08-44b8-8a86-f30abc234ff4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:22.840492 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-c1e23c7a-ef08-44b8-8a86-f30abc234ff4 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:42:22.840718 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-c1e23c7a-ef08-44b8-8a86-f30abc234ff4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:42:22.876617 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2fac2cc-50c3-42b4-951c-640c0c651f44 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:22.907876 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:22.920149 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c56360-e0f1-4e5e-8e8b-f5a96ac6e5f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:22.920846 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c56360-e0f1-4e5e-8e8b-f5a96ac6e5f5 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:22.921127 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c56360-e0f1-4e5e-8e8b-f5a96ac6e5f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:22.921423 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c56360-e0f1-4e5e-8e8b-f5a96ac6e5f5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:22.931974 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 325/666] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:42:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:42:22.937728 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e23c7a-ef08-44b8-8a86-f30abc234ff4 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:22.938968 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3c56360-e0f1-4e5e-8e8b-f5a96ac6e5f5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:22.940511 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3c56360-e0f1-4e5e-8e8b-f5a96ac6e5f5 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:22.962964 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-d3c56360-e0f1-4e5e-8e8b-f5a96ac6e5f5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:22.962964 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:22.962964 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:22.962964 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:22.962964 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:22.962964 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:22.962964 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:22.962964 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:22.962964 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:22.962964 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:22.962964 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:22.962964 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:22.962964 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:22.962964 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:22.962964 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:22.962964 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:22.964039 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:22.964039 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:22.964039 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:22.964039 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:22.964039 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:22.964039 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:22.964039 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:22.964039 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:22.964039 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:22.964039 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:22.964039 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:22.964039 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:22.964039 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:22.964039 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:22.964039 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:22.964039 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:22.964039 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:22.964932 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:22.964932 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:22.964932 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:22.966989 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 342/667] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:22.967969 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c56360-e0f1-4e5e-8e8b-f5a96ac6e5f5 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:22.987225 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:23.085995 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c56360-e0f1-4e5e-8e8b-f5a96ac6e5f5 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:23.119953 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e23c7a-ef08-44b8-8a86-f30abc234ff4 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:23.126101 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b5dfee4-a4d9-4617-afbe-d7de878051c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:23.126703 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5dfee4-a4d9-4617-afbe-d7de878051c7 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:23.126977 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5dfee4-a4d9-4617-afbe-d7de878051c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:23.127249 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5dfee4-a4d9-4617-afbe-d7de878051c7 None admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:23.127886 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b5dfee4-a4d9-4617-afbe-d7de878051c7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:23.128224 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b5dfee4-a4d9-4617-afbe-d7de878051c7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:23.129570 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b5dfee4-a4d9-4617-afbe-d7de878051c7 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:23.149628 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 343/668] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:23.150733 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b5dfee4-a4d9-4617-afbe-d7de878051c7 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:23.161718 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f199c5de-9456-4585-9b04-67d0a16799ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:23.162344 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f199c5de-9456-4585-9b04-67d0a16799ea None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:23.162618 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f199c5de-9456-4585-9b04-67d0a16799ea None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:23.162883 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f199c5de-9456-4585-9b04-67d0a16799ea None admin] PATH_INFO: `/v3/projects/service` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:23.170011 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f199c5de-9456-4585-9b04-67d0a16799ea None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:23.171516 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f199c5de-9456-4585-9b04-67d0a16799ea None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:23.184276 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-f199c5de-9456-4585-9b04-67d0a16799ea None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:42:23.184276 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:23.184276 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:23.184276 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:23.184276 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:23.184276 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:23.184276 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:23.184276 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:23.184276 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:23.184276 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:23.184276 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:23.184276 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:23.184276 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 00:42:23.184276 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 00:42:23.184276 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 00:42:23.185405 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 00:42:23.185405 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:23.185405 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:23.185405 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:23.185405 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:23.185405 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:23.185405 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:23.185405 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:23.185405 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:23.185405 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:23.185405 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:23.185405 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:23.185405 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:23.185405 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:23.185405 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:23.185405 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:23.185405 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:23.186315 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 00:42:23.186315 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 00:42:23.186315 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 00:42:23.186315 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 00:42:23.186315 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 00:42:23.186315 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 00:42:23.186315 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:42:23.186315 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:23.186315 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 326/669] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:42:22 2022] GET /identity/v3/projects/service => generated 88 bytes in 253 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:23.186958 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bcc3507-d90b-44d2-b5f4-d0058754c9fc None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:23.187239 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bcc3507-d90b-44d2-b5f4-d0058754c9fc None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:23.187510 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bcc3507-d90b-44d2-b5f4-d0058754c9fc None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:23.188887 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 327/670] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:42:23 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:42:23.197302 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bcc3507-d90b-44d2-b5f4-d0058754c9fc None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:23.247732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:23.258519 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:23.276356 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b5dfee4-a4d9-4617-afbe-d7de878051c7 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:23.317611 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb7be92b-dba6-4338-a641-d67196d4e149 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:23.318219 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7be92b-dba6-4338-a641-d67196d4e149 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:23.318495 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7be92b-dba6-4338-a641-d67196d4e149 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:23.318763 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7be92b-dba6-4338-a641-d67196d4e149 None admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:23.319407 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb7be92b-dba6-4338-a641-d67196d4e149 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:23.320394 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb7be92b-dba6-4338-a641-d67196d4e149 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:23.321886 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb7be92b-dba6-4338-a641-d67196d4e149 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:23.328206 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bcc3507-d90b-44d2-b5f4-d0058754c9fc None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:23.333653 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 344/671] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:23.334720 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb7be92b-dba6-4338-a641-d67196d4e149 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:23.366321 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc88010-8f37-4c11-903b-76f96bc68eff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:23.367826 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc88010-8f37-4c11-903b-76f96bc68eff None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:23.368102 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc88010-8f37-4c11-903b-76f96bc68eff None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:23.368367 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc88010-8f37-4c11-903b-76f96bc68eff None admin] PATH_INFO: `/v3/projects/service` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:23.375750 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc88010-8f37-4c11-903b-76f96bc68eff None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:23.377183 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc88010-8f37-4c11-903b-76f96bc68eff None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:23.389458 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-5fc88010-8f37-4c11-903b-76f96bc68eff None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:42:23.389458 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:23.389458 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:23.389458 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:23.389458 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:23.389458 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:23.389458 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:23.389458 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:23.389458 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:23.389458 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:23.389458 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:23.389458 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:23.389458 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 00:42:23.389458 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 00:42:23.389458 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 00:42:23.390461 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 00:42:23.390461 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:23.390461 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:23.390461 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:23.390461 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:23.390461 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:23.390461 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:23.390461 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:23.390461 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:23.390461 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:23.390461 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:23.390461 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:23.390461 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:23.390461 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:23.390461 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:23.390461 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:23.390461 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:23.391381 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 00:42:23.391381 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 00:42:23.391381 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 00:42:23.391381 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 00:42:23.391381 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 00:42:23.391381 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 00:42:23.391381 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:42:23.391381 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:23.391381 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 328/672] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:42:23 2022] GET /identity/v3/projects/service => generated 88 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:23.391878 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d490b1a5-4734-4bf7-ad72-af9465281480 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:23.392109 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d490b1a5-4734-4bf7-ad72-af9465281480 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:23.392410 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d490b1a5-4734-4bf7-ad72-af9465281480 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:23.412414 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-d490b1a5-4734-4bf7-ad72-af9465281480 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:42:23.413763 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-d490b1a5-4734-4bf7-ad72-af9465281480 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:42:23.465137 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb7be92b-dba6-4338-a641-d67196d4e149 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:23.492560 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 329/673] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:42:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:42:23.493451 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d490b1a5-4734-4bf7-ad72-af9465281480 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:23.512738 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a9d1dea-e454-4290-8067-f37515d94615 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:23.513650 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a9d1dea-e454-4290-8067-f37515d94615 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:23.514120 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a9d1dea-e454-4290-8067-f37515d94615 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:23.514525 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a9d1dea-e454-4290-8067-f37515d94615 None admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:23.515452 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a9d1dea-e454-4290-8067-f37515d94615 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:23.515867 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a9d1dea-e454-4290-8067-f37515d94615 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:23.517809 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a9d1dea-e454-4290-8067-f37515d94615 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:23.520986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:23.532224 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:23.540432 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 345/674] 10.222.0.52 () {56 vars in 1148 bytes} [Sun Sep 25 00:42:23 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:23.540432 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a9d1dea-e454-4290-8067-f37515d94615 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:23.619022 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d490b1a5-4734-4bf7-ad72-af9465281480 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:23.656325 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a9d1dea-e454-4290-8067-f37515d94615 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:23.665306 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b803e56-34e7-4a0b-a8c9-29e35be68f16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:23.665953 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b803e56-34e7-4a0b-a8c9-29e35be68f16 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:23.666229 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b803e56-34e7-4a0b-a8c9-29e35be68f16 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:23.666502 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b803e56-34e7-4a0b-a8c9-29e35be68f16 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:23.677392 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b803e56-34e7-4a0b-a8c9-29e35be68f16 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:42:23.682788 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-9b803e56-34e7-4a0b-a8c9-29e35be68f16 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 25 00:42:23.682788 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:23.682788 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:23.682788 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:23.682788 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:23.682788 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:23.682788 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:23.682788 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:23.682788 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:23.682788 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:23.682788 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:23.682788 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:23.682788 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:42:23.682788 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:42:23.682788 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:42:23.682788 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:42:23.683811 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:42:23.683811 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:42:23.683811 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:42:23.683811 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:42:23.683811 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:23.683811 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:23.683811 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:23.683811 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:23.683811 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:23.683811 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:23.683811 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:23.683811 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:23.683811 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:23.683811 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:23.683811 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:23.683811 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:23.684676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:23.684676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:23.684676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:23.684676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:23.684676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:42:23.684676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:42:23.684676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:42:23.684676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:42:23.684676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:42:23.684676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:42:23.684676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 25 00:42:23.684676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:23.684676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 330/675] 10.222.0.52 () {56 vars in 1102 bytes} [Sun Sep 25 00:42:23 2022] GET /identity/v3/roles/service => generated 85 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:23.684676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b803e56-34e7-4a0b-a8c9-29e35be68f16 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:23.704406 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a44860c2-1617-439d-94a3-6f6be142a8a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:23.705219 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44860c2-1617-439d-94a3-6f6be142a8a5 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:23.705520 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44860c2-1617-439d-94a3-6f6be142a8a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:23.706187 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44860c2-1617-439d-94a3-6f6be142a8a5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:23.717610 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a44860c2-1617-439d-94a3-6f6be142a8a5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:23.719039 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a44860c2-1617-439d-94a3-6f6be142a8a5 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:23.732011 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-a44860c2-1617-439d-94a3-6f6be142a8a5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:42:23.732011 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:23.732011 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:23.732011 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:23.732011 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:23.732011 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:23.732011 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:23.732011 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:23.732011 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:23.732011 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:23.732011 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:23.732011 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:23.732011 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 00:42:23.732011 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 00:42:23.732011 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 00:42:23.738986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 00:42:23.738986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:23.738986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:23.738986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:23.738986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:23.738986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:23.738986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:23.738986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:23.738986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:23.738986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:23.738986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:23.738986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:23.738986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:23.738986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:23.738986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:23.738986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:23.738986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:23.739997 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 00:42:23.739997 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 00:42:23.739997 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 00:42:23.739997 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 00:42:23.739997 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 00:42:23.739997 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 00:42:23.739997 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:42:23.739997 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:23.739997 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 346/676] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:42:23 2022] GET /identity/v3/projects/service => generated 88 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:23.739997 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a44860c2-1617-439d-94a3-6f6be142a8a5 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:23.801997 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:23.808397 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:23.819067 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b803e56-34e7-4a0b-a8c9-29e35be68f16 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:23.854001 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97022b11-a989-4877-9fb9-db0aa6894635 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:23.854605 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97022b11-a989-4877-9fb9-db0aa6894635 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:23.854872 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97022b11-a989-4877-9fb9-db0aa6894635 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:23.855134 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97022b11-a989-4877-9fb9-db0aa6894635 None admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:23.855820 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97022b11-a989-4877-9fb9-db0aa6894635 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:23.856117 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97022b11-a989-4877-9fb9-db0aa6894635 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:23.857569 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97022b11-a989-4877-9fb9-db0aa6894635 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:23.862019 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a44860c2-1617-439d-94a3-6f6be142a8a5 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:23.876266 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 331/677] 10.222.0.52 () {56 vars in 1148 bytes} [Sun Sep 25 00:42:23 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:23.877165 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97022b11-a989-4877-9fb9-db0aa6894635 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:23.907717 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee701ec4-53fd-4318-8530-c98aa51578bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:23.908325 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee701ec4-53fd-4318-8530-c98aa51578bf None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:23.908603 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee701ec4-53fd-4318-8530-c98aa51578bf None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:23.908872 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee701ec4-53fd-4318-8530-c98aa51578bf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:23.909631 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee701ec4-53fd-4318-8530-c98aa51578bf None admin] RBAC: Adding query filter params (role.id=a8e34e460e8a4ed7bcaa54da3c2d5a0c, scope.project.id=9db0fd07fc494203b2ada13f3cd046d3, user.id=4ebfc0abf1b842459a44ca9324252688) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:23.909939 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee701ec4-53fd-4318-8530-c98aa51578bf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:23.911364 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee701ec4-53fd-4318-8530-c98aa51578bf None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:23.918209 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 347/678] 10.222.0.52 () {56 vars in 1374 bytes} [Sun Sep 25 00:42:23 2022] GET /identity/v3/role_assignments?user.id=4ebfc0abf1b842459a44ca9324252688&scope.project.id=9db0fd07fc494203b2ada13f3cd046d3&role.id=a8e34e460e8a4ed7bcaa54da3c2d5a0c => generated 608 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:23.919031 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee701ec4-53fd-4318-8530-c98aa51578bf None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:24.013743 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97022b11-a989-4877-9fb9-db0aa6894635 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:24.057238 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f7292ab-0d0e-48b4-8f5f-de89c8d8be06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:24.057895 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7292ab-0d0e-48b4-8f5f-de89c8d8be06 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:24.058171 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7292ab-0d0e-48b4-8f5f-de89c8d8be06 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:24.058443 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7292ab-0d0e-48b4-8f5f-de89c8d8be06 None admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:24.059288 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f7292ab-0d0e-48b4-8f5f-de89c8d8be06 None admin] RBAC: Adding query filter params (name=service) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:24.059447 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f7292ab-0d0e-48b4-8f5f-de89c8d8be06 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:24.061300 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f7292ab-0d0e-48b4-8f5f-de89c8d8be06 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:24.075300 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 332/679] 10.222.0.52 () {56 vars in 1103 bytes} [Sun Sep 25 00:42:23 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:24.075300 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f7292ab-0d0e-48b4-8f5f-de89c8d8be06 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:24.088137 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee701ec4-53fd-4318-8530-c98aa51578bf None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:24.094777 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:24.109040 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:24.129181 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a8d7da7-3ff5-4e10-90e3-8333f6f1c2cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:24.129765 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8d7da7-3ff5-4e10-90e3-8333f6f1c2cb None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:24.130039 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8d7da7-3ff5-4e10-90e3-8333f6f1c2cb None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:24.130310 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8d7da7-3ff5-4e10-90e3-8333f6f1c2cb None admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:24.131002 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a8d7da7-3ff5-4e10-90e3-8333f6f1c2cb None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:24.131299 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a8d7da7-3ff5-4e10-90e3-8333f6f1c2cb None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:24.132979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a8d7da7-3ff5-4e10-90e3-8333f6f1c2cb None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:24.148312 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 348/680] 10.222.0.52 () {56 vars in 1148 bytes} [Sun Sep 25 00:42:23 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:24.149231 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a8d7da7-3ff5-4e10-90e3-8333f6f1c2cb None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:24.206987 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f7292ab-0d0e-48b4-8f5f-de89c8d8be06 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:24.243498 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee563ec-c374-4247-9fec-fd4b3498ef0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:24.244128 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee563ec-c374-4247-9fec-fd4b3498ef0b None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:24.244401 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee563ec-c374-4247-9fec-fd4b3498ef0b None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:24.244669 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee563ec-c374-4247-9fec-fd4b3498ef0b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:24.255656 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ee563ec-c374-4247-9fec-fd4b3498ef0b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:42:24.261240 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-8ee563ec-c374-4247-9fec-fd4b3498ef0b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:42:24.261240 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:24.261240 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:24.261240 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:24.261240 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:24.261240 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:24.261240 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:24.261240 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:24.261240 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:24.261240 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:24.261240 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:24.261240 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:24.261240 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:42:24.261240 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:42:24.261240 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:42:24.261240 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:42:24.262417 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:42:24.262417 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:42:24.262417 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:42:24.262417 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:42:24.262417 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:24.262417 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:24.262417 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:24.262417 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:24.262417 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:24.262417 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:24.262417 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:24.262417 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:24.262417 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:24.262417 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:24.262417 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:24.262417 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:24.263288 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:24.263288 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:24.263288 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:24.263288 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:24.263288 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:42:24.263288 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:42:24.263288 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:42:24.263288 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:42:24.263288 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:42:24.263288 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:42:24.263288 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:42:24.263288 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:24.263288 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 333/681] 10.222.0.52 () {56 vars in 1096 bytes} [Sun Sep 25 00:42:24 2022] GET /identity/v3/roles/admin => generated 83 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:24.263288 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee563ec-c374-4247-9fec-fd4b3498ef0b None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:24.277419 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a8d7da7-3ff5-4e10-90e3-8333f6f1c2cb None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:24.320726 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6342be7a-cc79-46ae-a95b-b8cbded64cdf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:24.321624 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6342be7a-cc79-46ae-a95b-b8cbded64cdf None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:24.322046 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6342be7a-cc79-46ae-a95b-b8cbded64cdf None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:24.322469 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6342be7a-cc79-46ae-a95b-b8cbded64cdf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:24.323631 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6342be7a-cc79-46ae-a95b-b8cbded64cdf None admin] RBAC: Adding query filter params (role.id=a8e34e460e8a4ed7bcaa54da3c2d5a0c, scope.project.id=9db0fd07fc494203b2ada13f3cd046d3, user.id=20c274e96dec478b98ef51f80ea893b0) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:24.324582 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6342be7a-cc79-46ae-a95b-b8cbded64cdf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:24.326618 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6342be7a-cc79-46ae-a95b-b8cbded64cdf None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:24.333669 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 349/682] 10.222.0.52 () {56 vars in 1374 bytes} [Sun Sep 25 00:42:24 2022] GET /identity/v3/role_assignments?user.id=20c274e96dec478b98ef51f80ea893b0&scope.project.id=9db0fd07fc494203b2ada13f3cd046d3&role.id=a8e34e460e8a4ed7bcaa54da3c2d5a0c => generated 608 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:24.334886 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6342be7a-cc79-46ae-a95b-b8cbded64cdf None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:24.360454 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:24.403092 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee563ec-c374-4247-9fec-fd4b3498ef0b None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:24.410379 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:24.451160 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d604f6f1-6f9c-40dc-9afc-1e35fe73b660 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:24.452180 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d604f6f1-6f9c-40dc-9afc-1e35fe73b660 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:24.452688 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d604f6f1-6f9c-40dc-9afc-1e35fe73b660 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:24.453194 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d604f6f1-6f9c-40dc-9afc-1e35fe73b660 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:24.467507 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d604f6f1-6f9c-40dc-9afc-1e35fe73b660 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:24.469496 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d604f6f1-6f9c-40dc-9afc-1e35fe73b660 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:24.490289 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-d604f6f1-6f9c-40dc-9afc-1e35fe73b660 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:24.490289 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:24.490289 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:24.490289 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:24.490289 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:24.490289 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:24.490289 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:24.490289 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:24.490289 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:24.490289 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:24.490289 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:24.490289 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:24.490289 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:24.490289 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:24.490289 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:24.490289 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:24.491762 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:24.491762 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:24.491762 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:24.491762 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:24.491762 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:24.491762 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:24.491762 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:24.491762 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:24.491762 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:24.491762 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:24.491762 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:24.491762 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:24.491762 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:24.491762 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:24.491762 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:24.491762 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:24.491762 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:24.492829 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:24.492829 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:24.492829 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:24.495321 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 334/683] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:24.496614 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d604f6f1-6f9c-40dc-9afc-1e35fe73b660 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:24.512130 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6342be7a-cc79-46ae-a95b-b8cbded64cdf None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:24.565764 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ee72d7-3e0a-42d0-afc4-a2dd84064b46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:24.566729 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ee72d7-3e0a-42d0-afc4-a2dd84064b46 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:24.567009 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ee72d7-3e0a-42d0-afc4-a2dd84064b46 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:24.567279 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ee72d7-3e0a-42d0-afc4-a2dd84064b46 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:24.568056 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ee72d7-3e0a-42d0-afc4-a2dd84064b46 None admin] RBAC: Adding query filter params (role.id=a8e34e460e8a4ed7bcaa54da3c2d5a0c, scope.project.id=9db0fd07fc494203b2ada13f3cd046d3, user.id=944f10f38ee2428a85599f6a714ac806) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:24.568362 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ee72d7-3e0a-42d0-afc4-a2dd84064b46 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:24.570546 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ee72d7-3e0a-42d0-afc4-a2dd84064b46 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:24.579278 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 350/684] 10.222.0.52 () {56 vars in 1374 bytes} [Sun Sep 25 00:42:24 2022] GET /identity/v3/role_assignments?user.id=944f10f38ee2428a85599f6a714ac806&scope.project.id=9db0fd07fc494203b2ada13f3cd046d3&role.id=a8e34e460e8a4ed7bcaa54da3c2d5a0c => generated 608 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:24.580155 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ee72d7-3e0a-42d0-afc4-a2dd84064b46 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:24.646737 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d604f6f1-6f9c-40dc-9afc-1e35fe73b660 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:24.685750 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:24.694040 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3374c823-0fa7-4711-a4bc-c1f2629d63aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:24.695266 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3374c823-0fa7-4711-a4bc-c1f2629d63aa None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:24.695556 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3374c823-0fa7-4711-a4bc-c1f2629d63aa None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:24.695827 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3374c823-0fa7-4711-a4bc-c1f2629d63aa None admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:24.696607 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3374c823-0fa7-4711-a4bc-c1f2629d63aa None admin] RBAC: Adding query filter params (name=admin) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:24.696954 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3374c823-0fa7-4711-a4bc-c1f2629d63aa None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:24.699795 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3374c823-0fa7-4711-a4bc-c1f2629d63aa None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:24.713011 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 335/685] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:42:24 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:24.719481 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3374c823-0fa7-4711-a4bc-c1f2629d63aa None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:24.727432 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ee72d7-3e0a-42d0-afc4-a2dd84064b46 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:24.740038 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:24.779464 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c678ccaa-49ed-44fe-9d65-d71816d27122 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:24.780244 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c678ccaa-49ed-44fe-9d65-d71816d27122 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:24.780631 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c678ccaa-49ed-44fe-9d65-d71816d27122 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:24.780999 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c678ccaa-49ed-44fe-9d65-d71816d27122 None admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:24.781789 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c678ccaa-49ed-44fe-9d65-d71816d27122 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:24.782198 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c678ccaa-49ed-44fe-9d65-d71816d27122 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:24.783707 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c678ccaa-49ed-44fe-9d65-d71816d27122 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:24.798731 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 351/686] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:24.804866 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c678ccaa-49ed-44fe-9d65-d71816d27122 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:24.870888 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3374c823-0fa7-4711-a4bc-c1f2629d63aa None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:24.910358 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0468e52-db20-4864-964e-2c4ee0859e44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:24.911090 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0468e52-db20-4864-964e-2c4ee0859e44 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:24.911473 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0468e52-db20-4864-964e-2c4ee0859e44 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:24.911849 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0468e52-db20-4864-964e-2c4ee0859e44 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:24.925862 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0468e52-db20-4864-964e-2c4ee0859e44 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:24.927479 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0468e52-db20-4864-964e-2c4ee0859e44 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:24.941924 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c678ccaa-49ed-44fe-9d65-d71816d27122 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:24.950067 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-a0468e52-db20-4864-964e-2c4ee0859e44 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:24.950067 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:24.950067 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:24.950067 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:24.950067 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:24.950067 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:24.950067 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:24.950067 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:24.950067 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:24.950067 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:24.950067 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:24.950067 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:24.950067 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:24.950067 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:24.950067 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:24.950067 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:24.951129 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:24.951129 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:24.951129 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:24.951129 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:24.951129 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:24.951129 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:24.951129 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:24.951129 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:24.951129 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:24.951129 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:24.951129 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:24.951129 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:24.951129 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:24.951129 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:24.951129 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:24.951129 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:24.951129 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:24.952036 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:24.952036 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:24.952036 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:24.952036 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 336/687] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:42:24.956716 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0468e52-db20-4864-964e-2c4ee0859e44 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:24.989824 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313fdfdf-9017-49bf-b3f0-00c6beef5574 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:24.990821 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313fdfdf-9017-49bf-b3f0-00c6beef5574 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:24.991295 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313fdfdf-9017-49bf-b3f0-00c6beef5574 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:24.992280 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313fdfdf-9017-49bf-b3f0-00c6beef5574 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:25.001236 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-313fdfdf-9017-49bf-b3f0-00c6beef5574 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:25.003309 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-313fdfdf-9017-49bf-b3f0-00c6beef5574 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:25.013279 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:25.018594 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-313fdfdf-9017-49bf-b3f0-00c6beef5574 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 25 00:42:25.018594 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:25.018594 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:25.018594 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:25.018594 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:25.018594 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:25.018594 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:25.018594 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:25.018594 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:25.018594 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:25.018594 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:25.018594 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:25.018594 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 00:42:25.018594 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 00:42:25.018594 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 00:42:25.018594 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 00:42:25.019609 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:25.019609 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:25.019609 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 00:42:25.019609 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:25.019609 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 00:42:25.019609 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:25.019609 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:25.019609 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:25.019609 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:25.019609 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:25.019609 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:25.019609 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:25.019609 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:25.019609 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:25.019609 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:25.019609 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:25.019609 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:25.019609 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:25.020561 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:25.020561 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:25.020561 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 00:42:25.020561 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 00:42:25.020561 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 00:42:25.020561 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 00:42:25.020561 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 00:42:25.020561 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 00:42:25.020561 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 25 00:42:25.020561 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:25.023746 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 352/688] 10.222.0.52 () {56 vars in 1096 bytes} [Sun Sep 25 00:42:24 2022] GET /identity/v3/users/swift => generated 83 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:42:25.033869 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313fdfdf-9017-49bf-b3f0-00c6beef5574 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:25.067095 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:25.107245 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0468e52-db20-4864-964e-2c4ee0859e44 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:25.172639 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313fdfdf-9017-49bf-b3f0-00c6beef5574 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:25.185875 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a941690d-d063-4c69-b029-b60ad97640c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:25.186888 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a941690d-d063-4c69-b029-b60ad97640c3 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:25.187371 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a941690d-d063-4c69-b029-b60ad97640c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:25.187851 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a941690d-d063-4c69-b029-b60ad97640c3 None admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:25.188725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a941690d-d063-4c69-b029-b60ad97640c3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:25.189225 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a941690d-d063-4c69-b029-b60ad97640c3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:25.190885 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a941690d-d063-4c69-b029-b60ad97640c3 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:25.205778 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 337/689] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:25.211848 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a941690d-d063-4c69-b029-b60ad97640c3 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:25.244157 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deced3c1-e1a9-4743-9cc5-9f58386735d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:25.245528 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deced3c1-e1a9-4743-9cc5-9f58386735d9 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:25.246900 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deced3c1-e1a9-4743-9cc5-9f58386735d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:25.247381 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deced3c1-e1a9-4743-9cc5-9f58386735d9 None admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:25.248484 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deced3c1-e1a9-4743-9cc5-9f58386735d9 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:25.249936 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deced3c1-e1a9-4743-9cc5-9f58386735d9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:25.253464 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deced3c1-e1a9-4743-9cc5-9f58386735d9 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:25.293644 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 353/690] 10.222.0.52 () {56 vars in 1135 bytes} [Sun Sep 25 00:42:25 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:25.301466 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deced3c1-e1a9-4743-9cc5-9f58386735d9 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:25.436080 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:25.446605 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:25.450784 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a941690d-d063-4c69-b029-b60ad97640c3 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:25.468206 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deced3c1-e1a9-4743-9cc5-9f58386735d9 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:25.549984 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27958df2-a6af-48e4-906d-32858d1fefb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:25.551030 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27958df2-a6af-48e4-906d-32858d1fefb3 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:25.551512 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27958df2-a6af-48e4-906d-32858d1fefb3 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:25.551986 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27958df2-a6af-48e4-906d-32858d1fefb3 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:25.567977 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27958df2-a6af-48e4-906d-32858d1fefb3 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:25.570725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27958df2-a6af-48e4-906d-32858d1fefb3 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:25.571765 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7599fd66-c578-4671-bda4-2263fa04f328 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:25.572390 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7599fd66-c578-4671-bda4-2263fa04f328 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:25.572667 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7599fd66-c578-4671-bda4-2263fa04f328 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:25.572937 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7599fd66-c578-4671-bda4-2263fa04f328 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:25.596749 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-27958df2-a6af-48e4-906d-32858d1fefb3 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 25 00:42:25.596749 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:25.596749 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:25.596749 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:25.596749 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:25.596749 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:25.596749 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:25.596749 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:25.596749 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:25.596749 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:25.596749 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:25.596749 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:25.596749 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 00:42:25.596749 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 00:42:25.596749 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 00:42:25.596749 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 00:42:25.597878 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:25.597878 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:25.597878 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 00:42:25.597878 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:25.597878 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 00:42:25.597878 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:25.597878 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:25.597878 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:25.597878 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:25.597878 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:25.597878 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:25.597878 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:25.597878 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:25.597878 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:25.597878 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:25.597878 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:25.597878 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:25.597878 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:25.598834 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:25.598834 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:25.598834 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 00:42:25.598834 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 00:42:25.598834 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 00:42:25.598834 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 00:42:25.598834 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 00:42:25.598834 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 00:42:25.598834 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 25 00:42:25.598834 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:25.600901 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 338/691] 10.222.0.52 () {56 vars in 1093 bytes} [Sun Sep 25 00:42:25 2022] GET /identity/v3/users/nova => generated 82 bytes in 390 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:25.607048 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27958df2-a6af-48e4-906d-32858d1fefb3 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:25.704647 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7599fd66-c578-4671-bda4-2263fa04f328 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:25.706616 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7599fd66-c578-4671-bda4-2263fa04f328 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:25.793221 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27958df2-a6af-48e4-906d-32858d1fefb3 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:25.835717 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:25.867397 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 354/692] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:42:25 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 565 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 00:42:25.873082 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7599fd66-c578-4671-bda4-2263fa04f328 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:25.883962 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90eea7e4-579f-4f50-aa14-8c97c3f48196 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:25.884605 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90eea7e4-579f-4f50-aa14-8c97c3f48196 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:25.884848 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90eea7e4-579f-4f50-aa14-8c97c3f48196 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:25.885113 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90eea7e4-579f-4f50-aa14-8c97c3f48196 None admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:25.885868 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90eea7e4-579f-4f50-aa14-8c97c3f48196 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:25.886171 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90eea7e4-579f-4f50-aa14-8c97c3f48196 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:25.887595 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90eea7e4-579f-4f50-aa14-8c97c3f48196 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:25.911671 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 339/693] 10.222.0.52 () {56 vars in 1133 bytes} [Sun Sep 25 00:42:25 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 305 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:42:25.917722 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90eea7e4-579f-4f50-aa14-8c97c3f48196 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:25.939236 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:26.059661 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7599fd66-c578-4671-bda4-2263fa04f328 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:26.065758 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90eea7e4-579f-4f50-aa14-8c97c3f48196 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:26.136458 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193947e3-23ac-4dc2-9e5d-1db9e770cc51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:26.137096 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193947e3-23ac-4dc2-9e5d-1db9e770cc51 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:26.137401 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193947e3-23ac-4dc2-9e5d-1db9e770cc51 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:26.139598 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e032fd66-eb49-49cf-80cc-c0e8c7709c92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:26.140213 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e032fd66-eb49-49cf-80cc-c0e8c7709c92 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:26.140488 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e032fd66-eb49-49cf-80cc-c0e8c7709c92 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:26.140757 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e032fd66-eb49-49cf-80cc-c0e8c7709c92 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:26.141635 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193947e3-23ac-4dc2-9e5d-1db9e770cc51 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:26.164291 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-193947e3-23ac-4dc2-9e5d-1db9e770cc51 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:26.166278 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-193947e3-23ac-4dc2-9e5d-1db9e770cc51 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:26.197491 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-193947e3-23ac-4dc2-9e5d-1db9e770cc51 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:26.197491 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:26.197491 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:26.197491 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:26.197491 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:26.197491 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:26.197491 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:26.197491 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:26.197491 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:26.197491 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:26.197491 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:26.197491 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:26.197491 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:26.197491 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:26.197491 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:26.197491 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:26.198536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:26.198536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:26.198536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:26.198536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:26.198536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:26.198536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:26.198536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:26.198536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:26.198536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:26.198536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:26.198536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:26.198536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:26.198536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:26.198536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:26.198536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:26.198536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:26.198536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:26.199488 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:26.199488 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:26.199488 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:26.200666 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 355/694] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 327 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:26.206858 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193947e3-23ac-4dc2-9e5d-1db9e770cc51 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:26.267505 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e032fd66-eb49-49cf-80cc-c0e8c7709c92 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:26.269035 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e032fd66-eb49-49cf-80cc-c0e8c7709c92 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:26.277959 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:26.292852 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:26.372036 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193947e3-23ac-4dc2-9e5d-1db9e770cc51 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:26.412030 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2573ff7-6ea5-4874-a413-687764cf6b8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:26.412634 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2573ff7-6ea5-4874-a413-687764cf6b8f None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:26.412906 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2573ff7-6ea5-4874-a413-687764cf6b8f None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:26.413348 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 340/695] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:42:25 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 496 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 00:42:26.413718 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2573ff7-6ea5-4874-a413-687764cf6b8f None admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:26.414695 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49cd585-4046-437d-86bf-f15c8ca665f7 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:26.414809 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2573ff7-6ea5-4874-a413-687764cf6b8f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:26.414992 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49cd585-4046-437d-86bf-f15c8ca665f7 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:26.415086 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2573ff7-6ea5-4874-a413-687764cf6b8f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:26.415260 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49cd585-4046-437d-86bf-f15c8ca665f7 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:26.416437 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2573ff7-6ea5-4874-a413-687764cf6b8f None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:26.416809 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 341/696] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:42:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:42:26.422584 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d49cd585-4046-437d-86bf-f15c8ca665f7 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:26.435375 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 356/697] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:42:26.436784 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f63937-e366-4b3a-8c50-2b0756c8fd30 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:26.437060 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f63937-e366-4b3a-8c50-2b0756c8fd30 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:26.437364 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f63937-e366-4b3a-8c50-2b0756c8fd30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:26.457830 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-a1f63937-e366-4b3a-8c50-2b0756c8fd30 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:42:26.459169 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-a1f63937-e366-4b3a-8c50-2b0756c8fd30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:42:26.537806 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 357/698] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:42:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:42:26.539390 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f63937-e366-4b3a-8c50-2b0756c8fd30 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:26.545821 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d49cd585-4046-437d-86bf-f15c8ca665f7 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:26.567646 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:26.572921 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:26.589879 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfd3d4ef-83c6-4dd3-ac64-aff38baf96fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:26.590512 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd3d4ef-83c6-4dd3-ac64-aff38baf96fb None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:26.590789 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd3d4ef-83c6-4dd3-ac64-aff38baf96fb None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:26.591061 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd3d4ef-83c6-4dd3-ac64-aff38baf96fb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:26.602587 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfd3d4ef-83c6-4dd3-ac64-aff38baf96fb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:26.604047 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfd3d4ef-83c6-4dd3-ac64-aff38baf96fb None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:26.620614 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-dfd3d4ef-83c6-4dd3-ac64-aff38baf96fb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:26.620614 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:26.620614 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:26.620614 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:26.620614 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:26.620614 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:26.620614 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:26.620614 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:26.620614 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:26.620614 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:26.620614 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:26.620614 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:26.620614 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:26.620614 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:26.620614 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:26.620614 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:26.622678 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:26.622678 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:26.622678 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:26.622678 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:26.622678 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:26.622678 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:26.622678 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:26.622678 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:26.622678 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:26.622678 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:26.622678 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:26.622678 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:26.622678 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:26.622678 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:26.622678 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:26.622678 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:26.622678 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:26.623836 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:26.623836 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:26.623836 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:26.623836 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 342/699] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:26.629627 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfd3d4ef-83c6-4dd3-ac64-aff38baf96fb None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:26.755791 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f63937-e366-4b3a-8c50-2b0756c8fd30 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:26.757760 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfd3d4ef-83c6-4dd3-ac64-aff38baf96fb None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:26.795868 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1feebe83-4f5b-47c8-bd81-9562ff75984f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:26.796681 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1feebe83-4f5b-47c8-bd81-9562ff75984f None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:26.797066 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1feebe83-4f5b-47c8-bd81-9562ff75984f None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:26.797481 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1feebe83-4f5b-47c8-bd81-9562ff75984f None admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:26.798345 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1feebe83-4f5b-47c8-bd81-9562ff75984f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:26.798797 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1feebe83-4f5b-47c8-bd81-9562ff75984f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:26.800369 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1feebe83-4f5b-47c8-bd81-9562ff75984f None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:26.814064 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 343/700] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:26.822777 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1feebe83-4f5b-47c8-bd81-9562ff75984f None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:26.858739 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:26.860844 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03092437-3c55-49e7-969b-9f8b04a56933 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:26.861687 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03092437-3c55-49e7-969b-9f8b04a56933 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:26.861868 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03092437-3c55-49e7-969b-9f8b04a56933 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:26.862136 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03092437-3c55-49e7-969b-9f8b04a56933 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:26.882913 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03092437-3c55-49e7-969b-9f8b04a56933 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:26.884482 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03092437-3c55-49e7-969b-9f8b04a56933 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:26.900626 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-03092437-3c55-49e7-969b-9f8b04a56933 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:42:26.900626 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:26.900626 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:26.900626 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:26.900626 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:26.900626 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:26.900626 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:26.900626 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:26.900626 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:26.900626 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:26.900626 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:26.900626 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:26.900626 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 00:42:26.900626 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 00:42:26.900626 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 00:42:26.901659 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 00:42:26.901659 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:26.901659 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:26.901659 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:26.901659 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:26.901659 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:26.901659 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:26.901659 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:26.901659 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:26.901659 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:26.901659 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:26.901659 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:26.901659 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:26.901659 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:26.901659 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:26.901659 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:26.901659 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:26.902591 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 00:42:26.902591 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 00:42:26.902591 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 00:42:26.902591 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 00:42:26.902591 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 00:42:26.902591 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 00:42:26.902591 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:42:26.902591 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:26.902591 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 358/701] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:42:26 2022] GET /identity/v3/projects/service => generated 88 bytes in 364 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:26.903108 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03092437-3c55-49e7-969b-9f8b04a56933 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:26.940181 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1feebe83-4f5b-47c8-bd81-9562ff75984f None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:26.979375 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d029fd0-cb44-45cd-993b-ad4ebacdde66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:26.980014 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d029fd0-cb44-45cd-993b-ad4ebacdde66 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:26.980446 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d029fd0-cb44-45cd-993b-ad4ebacdde66 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:26.980721 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d029fd0-cb44-45cd-993b-ad4ebacdde66 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:26.988389 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d029fd0-cb44-45cd-993b-ad4ebacdde66 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:26.990085 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d029fd0-cb44-45cd-993b-ad4ebacdde66 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:26.997409 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:27.003888 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-9d029fd0-cb44-45cd-993b-ad4ebacdde66 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:42:27.003888 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:27.003888 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:27.003888 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:27.003888 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:27.003888 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:27.003888 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:27.003888 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:27.003888 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:27.003888 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:27.003888 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:27.003888 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:27.003888 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 00:42:27.003888 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 00:42:27.003888 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 00:42:27.005718 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 00:42:27.005718 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:27.005718 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:27.005718 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:27.005718 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:27.005718 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:27.005718 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:27.005718 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:27.005718 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:27.005718 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:27.005718 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:27.005718 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:27.005718 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:27.005718 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:27.005718 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:27.005718 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:27.005718 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:27.008701 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 00:42:27.008701 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 00:42:27.008701 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 00:42:27.008701 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 00:42:27.008701 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 00:42:27.008701 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 00:42:27.008701 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:42:27.008701 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:27.008701 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 344/702] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:42:26 2022] GET /identity/v3/projects/service => generated 88 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:27.008701 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d029fd0-cb44-45cd-993b-ad4ebacdde66 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:27.017313 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03092437-3c55-49e7-969b-9f8b04a56933 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:27.054775 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aaa2c75-04cf-4b63-8dc9-06836d5c6492 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:27.055501 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aaa2c75-04cf-4b63-8dc9-06836d5c6492 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:27.055778 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aaa2c75-04cf-4b63-8dc9-06836d5c6492 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:27.056474 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aaa2c75-04cf-4b63-8dc9-06836d5c6492 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:27.067682 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aaa2c75-04cf-4b63-8dc9-06836d5c6492 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:27.069141 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aaa2c75-04cf-4b63-8dc9-06836d5c6492 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:27.086108 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-6aaa2c75-04cf-4b63-8dc9-06836d5c6492 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:27.086108 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:27.086108 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:27.086108 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:27.086108 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:27.086108 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:27.086108 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:27.086108 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:27.086108 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:27.086108 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:27.086108 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:27.086108 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:27.086108 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:27.086108 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:27.086108 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:27.086108 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:27.087362 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:27.087362 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:27.087362 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:27.087362 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:27.087362 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:27.087362 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:27.087362 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:27.087362 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:27.087362 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:27.087362 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:27.087362 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:27.087362 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:27.087362 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:27.087362 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:27.087362 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:27.087362 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:27.087362 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:27.088344 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:27.088344 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:27.088344 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:27.092395 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 359/703] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:27.093787 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aaa2c75-04cf-4b63-8dc9-06836d5c6492 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:27.116974 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:27.129531 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d029fd0-cb44-45cd-993b-ad4ebacdde66 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:27.171195 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6ea526b-7dee-47c8-a896-1f3537418e8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:27.172036 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ea526b-7dee-47c8-a896-1f3537418e8a None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:27.172531 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ea526b-7dee-47c8-a896-1f3537418e8a None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:27.173042 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ea526b-7dee-47c8-a896-1f3537418e8a None admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:27.174020 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6ea526b-7dee-47c8-a896-1f3537418e8a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:27.174469 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6ea526b-7dee-47c8-a896-1f3537418e8a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:27.176054 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6ea526b-7dee-47c8-a896-1f3537418e8a None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:27.190125 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 345/704] 10.222.0.52 () {56 vars in 1148 bytes} [Sun Sep 25 00:42:27 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:27.191552 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6ea526b-7dee-47c8-a896-1f3537418e8a None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:27.211884 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aaa2c75-04cf-4b63-8dc9-06836d5c6492 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:27.250058 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610a8d70-db74-494b-9efe-40bc7191ecab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:27.250680 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610a8d70-db74-494b-9efe-40bc7191ecab None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:27.250955 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610a8d70-db74-494b-9efe-40bc7191ecab None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:27.251221 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610a8d70-db74-494b-9efe-40bc7191ecab None admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:27.251924 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-610a8d70-db74-494b-9efe-40bc7191ecab None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:27.252226 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-610a8d70-db74-494b-9efe-40bc7191ecab None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:27.253775 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-610a8d70-db74-494b-9efe-40bc7191ecab None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:27.261786 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:27.267241 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 360/705] 10.222.0.52 () {56 vars in 1148 bytes} [Sun Sep 25 00:42:27 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:27.268406 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610a8d70-db74-494b-9efe-40bc7191ecab None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:27.311776 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6ea526b-7dee-47c8-a896-1f3537418e8a None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:27.348691 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f495bc-8092-4d14-af04-c91204aca8f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:27.349333 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f495bc-8092-4d14-af04-c91204aca8f8 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:27.349617 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f495bc-8092-4d14-af04-c91204aca8f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:27.349881 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f495bc-8092-4d14-af04-c91204aca8f8 None admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:27.350516 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9f495bc-8092-4d14-af04-c91204aca8f8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:27.350939 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9f495bc-8092-4d14-af04-c91204aca8f8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:27.352759 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9f495bc-8092-4d14-af04-c91204aca8f8 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:27.365602 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 346/706] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:27.367736 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13f1b0f-f73d-48d4-8f44-bdaf020c4353 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:27.368253 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13f1b0f-f73d-48d4-8f44-bdaf020c4353 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:27.368735 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13f1b0f-f73d-48d4-8f44-bdaf020c4353 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:27.370471 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 347/707] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:42:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:42:27.371416 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13f1b0f-f73d-48d4-8f44-bdaf020c4353 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:27.383322 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610a8d70-db74-494b-9efe-40bc7191ecab None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:27.383935 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:27.421322 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-658ecb1d-b613-4cdb-bf39-84725e9de213 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:27.421839 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658ecb1d-b613-4cdb-bf39-84725e9de213 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:27.422109 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658ecb1d-b613-4cdb-bf39-84725e9de213 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:27.422371 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658ecb1d-b613-4cdb-bf39-84725e9de213 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:27.423081 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-658ecb1d-b613-4cdb-bf39-84725e9de213 None admin] RBAC: Adding query filter params (role.id=a8e34e460e8a4ed7bcaa54da3c2d5a0c, scope.project.id=9db0fd07fc494203b2ada13f3cd046d3, user.id=da606b3906dd4171bd76fd281e03e258) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:27.423382 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-658ecb1d-b613-4cdb-bf39-84725e9de213 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:27.424805 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-658ecb1d-b613-4cdb-bf39-84725e9de213 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:27.437305 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 361/708] 10.222.0.52 () {56 vars in 1374 bytes} [Sun Sep 25 00:42:27 2022] GET /identity/v3/role_assignments?user.id=da606b3906dd4171bd76fd281e03e258&scope.project.id=9db0fd07fc494203b2ada13f3cd046d3&role.id=a8e34e460e8a4ed7bcaa54da3c2d5a0c => generated 608 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:27.437305 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-658ecb1d-b613-4cdb-bf39-84725e9de213 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:27.506322 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:27.512113 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13f1b0f-f73d-48d4-8f44-bdaf020c4353 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:27.555489 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-658ecb1d-b613-4cdb-bf39-84725e9de213 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:27.558015 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81bd646c-44a9-4a44-bbd0-769973571537 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:27.558971 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81bd646c-44a9-4a44-bbd0-769973571537 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:27.559393 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81bd646c-44a9-4a44-bbd0-769973571537 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:27.559798 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81bd646c-44a9-4a44-bbd0-769973571537 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:27.560680 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81bd646c-44a9-4a44-bbd0-769973571537 None admin] RBAC: Adding query filter params (role.id=2c242abf6ed8410ab0a4a354ca1794fb, scope.project.id=9db0fd07fc494203b2ada13f3cd046d3, user.id=f142988a4dd14a5f96eacea45c046f85) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:27.561131 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81bd646c-44a9-4a44-bbd0-769973571537 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:27.562795 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81bd646c-44a9-4a44-bbd0-769973571537 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:27.570771 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 348/709] 10.222.0.52 () {56 vars in 1374 bytes} [Sun Sep 25 00:42:27 2022] GET /identity/v3/role_assignments?user.id=f142988a4dd14a5f96eacea45c046f85&scope.project.id=9db0fd07fc494203b2ada13f3cd046d3&role.id=2c242abf6ed8410ab0a4a354ca1794fb => generated 259 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:27.572560 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e3ba597-1b17-46d6-b4d3-df4e63865443 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:27.573117 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e3ba597-1b17-46d6-b4d3-df4e63865443 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:27.573611 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e3ba597-1b17-46d6-b4d3-df4e63865443 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:27.594177 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-5e3ba597-1b17-46d6-b4d3-df4e63865443 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:42:27.595956 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-5e3ba597-1b17-46d6-b4d3-df4e63865443 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:42:27.596401 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05435e23-2376-4a8a-8df2-45449776a91b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:27.597080 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05435e23-2376-4a8a-8df2-45449776a91b None admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:27.597405 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05435e23-2376-4a8a-8df2-45449776a91b None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:27.597684 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05435e23-2376-4a8a-8df2-45449776a91b None admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:27.598442 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05435e23-2376-4a8a-8df2-45449776a91b None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:27.600079 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05435e23-2376-4a8a-8df2-45449776a91b None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:27.628096 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 362/710] 10.222.0.52 () {60 vars in 1132 bytes} [Sun Sep 25 00:42:27 2022] POST /identity/v3/users => generated 254 bytes in 196 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:42:27.657915 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41cdb7f3-325c-4914-9848-35ad8594fe4b None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:27.658295 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41cdb7f3-325c-4914-9848-35ad8594fe4b None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:27.658458 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41cdb7f3-325c-4914-9848-35ad8594fe4b None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:27.660145 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 363/711] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:42:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 00:42:27.666604 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed056c8e-a5f2-4813-93b4-01951d7c8fe9 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:27.666928 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed056c8e-a5f2-4813-93b4-01951d7c8fe9 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:27.667205 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed056c8e-a5f2-4813-93b4-01951d7c8fe9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:27.690087 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-ed056c8e-a5f2-4813-93b4-01951d7c8fe9 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:42:27.691402 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-ed056c8e-a5f2-4813-93b4-01951d7c8fe9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:42:27.706395 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 349/712] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:42:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:42:27.782401 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 364/713] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:42:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 25 00:42:27.994602 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e3ba597-1b17-46d6-b4d3-df4e63865443 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:27.997868 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:28.118226 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e3ba597-1b17-46d6-b4d3-df4e63865443 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:28.126403 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed056c8e-a5f2-4813-93b4-01951d7c8fe9 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:28.186691 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e9ccc8-1518-44e8-98b0-45a7fa8897f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:28.187665 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e9ccc8-1518-44e8-98b0-45a7fa8897f7 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:28.188187 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e9ccc8-1518-44e8-98b0-45a7fa8897f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:28.188665 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e9ccc8-1518-44e8-98b0-45a7fa8897f7 None admin] PATH_INFO: `/v3/services/network` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:28.189500 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e9ccc8-1518-44e8-98b0-45a7fa8897f7 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:28.191089 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e9ccc8-1518-44e8-98b0-45a7fa8897f7 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:28.205616 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-b3e9ccc8-1518-44e8-98b0-45a7fa8897f7 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 25 00:42:28.205616 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:28.205616 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 00:42:28.205616 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 00:42:28.205616 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 00:42:28.205616 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 00:42:28.205616 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 00:42:28.205616 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:42:28.205616 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 25 00:42:28.205616 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:28.205616 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 00:42:28.205616 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:28.205616 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:28.205616 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:28.205616 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:28.205616 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:28.205616 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:28.206824 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:28.206824 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:28.206824 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:28.206824 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:28.206824 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:28.206824 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:28.206824 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 00:42:28.206824 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 00:42:28.206824 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 00:42:28.206824 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 00:42:28.206824 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:28.206824 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:28.206824 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:28.206824 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:28.206824 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:28.206824 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:28.207671 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:28.207671 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:28.207671 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:28.207671 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:28.207671 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:28.207671 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:28.207671 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:28.207671 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:28.207671 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:28.207671 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:28.207671 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 00:42:28.207671 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:42:28.207671 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 25 00:42:28.207671 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:28.209421 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:28.217620 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 350/714] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:42:27 2022] GET /identity/v3/services/network => generated 88 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:28.219824 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e9ccc8-1518-44e8-98b0-45a7fa8897f7 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:28.320115 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed056c8e-a5f2-4813-93b4-01951d7c8fe9 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:28.344832 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:28.377356 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c460f71f-6de1-4fba-b131-a6ce06f50fdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:28.378567 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c460f71f-6de1-4fba-b131-a6ce06f50fdd None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:28.379838 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c460f71f-6de1-4fba-b131-a6ce06f50fdd None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:28.380482 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c460f71f-6de1-4fba-b131-a6ce06f50fdd None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:28.383255 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c460f71f-6de1-4fba-b131-a6ce06f50fdd None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:28.385090 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c460f71f-6de1-4fba-b131-a6ce06f50fdd None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:28.405326 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-c460f71f-6de1-4fba-b131-a6ce06f50fdd None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 25 00:42:28.405326 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:28.405326 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 00:42:28.405326 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 00:42:28.405326 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 00:42:28.405326 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 00:42:28.405326 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 00:42:28.405326 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:42:28.405326 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 25 00:42:28.405326 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:28.405326 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 00:42:28.405326 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:28.405326 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:28.405326 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:28.405326 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:28.405326 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:28.405326 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:28.406407 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:28.406407 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:28.406407 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:28.406407 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:28.406407 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:28.406407 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:28.406407 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 00:42:28.406407 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 00:42:28.406407 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 00:42:28.406407 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 00:42:28.406407 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:28.406407 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:28.406407 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:28.406407 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:28.406407 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:28.406407 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:28.407241 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:28.407241 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:28.407241 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:28.407241 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:28.407241 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:28.407241 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:28.407241 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:28.407241 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:28.407241 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:28.407241 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:28.407241 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 00:42:28.407241 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:42:28.407241 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 25 00:42:28.407241 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:28.415041 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 365/715] 10.222.0.52 () {56 vars in 1129 bytes} [Sun Sep 25 00:42:28 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 289 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:42:28.419307 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e9ccc8-1518-44e8-98b0-45a7fa8897f7 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:28.422054 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c460f71f-6de1-4fba-b131-a6ce06f50fdd None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:28.498941 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24359030-f2ce-467b-b597-dce4ca566335 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:28.499562 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24359030-f2ce-467b-b597-dce4ca566335 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:28.499837 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24359030-f2ce-467b-b597-dce4ca566335 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:28.500103 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24359030-f2ce-467b-b597-dce4ca566335 None admin] PATH_INFO: `/v3/services` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:28.500742 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24359030-f2ce-467b-b597-dce4ca566335 None admin] RBAC: Adding query filter params (name=network) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:28.501044 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24359030-f2ce-467b-b597-dce4ca566335 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:28.502461 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24359030-f2ce-467b-b597-dce4ca566335 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:28.516586 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 351/716] 10.222.0.52 () {56 vars in 1112 bytes} [Sun Sep 25 00:42:28 2022] GET /identity/v3/services?name=network => generated 124 bytes in 297 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:28.532515 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24359030-f2ce-467b-b597-dce4ca566335 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:28.556596 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c460f71f-6de1-4fba-b131-a6ce06f50fdd None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:28.571888 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:28.596051 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af92dbc-210d-415f-9522-63964804e33a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:28.598302 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af92dbc-210d-415f-9522-63964804e33a None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:28.598801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af92dbc-210d-415f-9522-63964804e33a None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:28.599483 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af92dbc-210d-415f-9522-63964804e33a None admin] PATH_INFO: `/v3/services` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:28.601035 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1af92dbc-210d-415f-9522-63964804e33a None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:28.601783 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1af92dbc-210d-415f-9522-63964804e33a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:28.604001 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1af92dbc-210d-415f-9522-63964804e33a None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:28.618025 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 366/717] 10.222.0.52 () {56 vars in 1124 bytes} [Sun Sep 25 00:42:28 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:28.630311 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af92dbc-210d-415f-9522-63964804e33a None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:28.725850 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24359030-f2ce-467b-b597-dce4ca566335 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:28.747026 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af92dbc-210d-415f-9522-63964804e33a None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:28.783615 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16cf95e1-7617-46f6-970b-ef5f5a0c7607 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:28.784575 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16cf95e1-7617-46f6-970b-ef5f5a0c7607 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:28.785079 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16cf95e1-7617-46f6-970b-ef5f5a0c7607 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:28.785572 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16cf95e1-7617-46f6-970b-ef5f5a0c7607 None admin] PATH_INFO: `/v3/services` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:28.786424 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16cf95e1-7617-46f6-970b-ef5f5a0c7607 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:28.786926 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16cf95e1-7617-46f6-970b-ef5f5a0c7607 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:28.788559 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16cf95e1-7617-46f6-970b-ef5f5a0c7607 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:28.788751 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:28.796438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 367/718] 10.222.0.52 () {56 vars in 1124 bytes} [Sun Sep 25 00:42:28 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:28.805286 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ecf0096-6c33-48c6-b318-29efe5d62cc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:28.805286 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ecf0096-6c33-48c6-b318-29efe5d62cc3 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:28.805286 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ecf0096-6c33-48c6-b318-29efe5d62cc3 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:28.805286 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ecf0096-6c33-48c6-b318-29efe5d62cc3 None admin] PATH_INFO: `/v3/services` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:28.805679 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ecf0096-6c33-48c6-b318-29efe5d62cc3 None admin] RBAC: Adding query filter params (type=network) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:28.805679 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ecf0096-6c33-48c6-b318-29efe5d62cc3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:28.805679 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ecf0096-6c33-48c6-b318-29efe5d62cc3 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:28.816764 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 352/719] 10.222.0.52 () {56 vars in 1112 bytes} [Sun Sep 25 00:42:28 2022] GET /identity/v3/services?type=network => generated 124 bytes in 285 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:42:30.301942 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2238a103-a893-4a75-91f7-b776a7ffe304 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:30.302985 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2238a103-a893-4a75-91f7-b776a7ffe304 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:30.303507 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2238a103-a893-4a75-91f7-b776a7ffe304 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:30.306111 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 368/720] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:42:30 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:42:30.311383 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4225a2d9-60b9-4a9d-9be1-602e5d3423ae None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:30.312476 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4225a2d9-60b9-4a9d-9be1-602e5d3423ae None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:30.312953 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4225a2d9-60b9-4a9d-9be1-602e5d3423ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:30.342650 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-4225a2d9-60b9-4a9d-9be1-602e5d3423ae None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:42:30.344410 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-4225a2d9-60b9-4a9d-9be1-602e5d3423ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:42:30.347326 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1896b4-d4bb-4192-8cea-9c10278dbec5 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:30.347894 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1896b4-d4bb-4192-8cea-9c10278dbec5 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:30.348366 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1896b4-d4bb-4192-8cea-9c10278dbec5 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:30.349965 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 369/721] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:42:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:42:30.356995 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbad9066-463f-4d74-b94f-ae7e9ede09a5 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:30.357364 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbad9066-463f-4d74-b94f-ae7e9ede09a5 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:30.357567 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbad9066-463f-4d74-b94f-ae7e9ede09a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:30.379003 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-fbad9066-463f-4d74-b94f-ae7e9ede09a5 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:42:30.380569 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-fbad9066-463f-4d74-b94f-ae7e9ede09a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:42:30.422130 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:30.463729 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 370/722] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:42:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:42:30.469876 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 353/723] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:42:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:42:30.564612 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634bfe74-9695-47d7-895f-ebb29e748d29 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:30.565221 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634bfe74-9695-47d7-895f-ebb29e748d29 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:30.565756 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634bfe74-9695-47d7-895f-ebb29e748d29 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:30.567526 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 371/724] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:42:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:42:30.574034 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6172eec6-fba8-4595-b28e-a7dba7512dbd None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:30.574567 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6172eec6-fba8-4595-b28e-a7dba7512dbd None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:30.575082 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6172eec6-fba8-4595-b28e-a7dba7512dbd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:30.594191 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-6172eec6-fba8-4595-b28e-a7dba7512dbd None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:42:30.595739 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-6172eec6-fba8-4595-b28e-a7dba7512dbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:42:30.687001 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 354/725] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:42:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:42:30.804775 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634bfe74-9695-47d7-895f-ebb29e748d29 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:30.914513 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6172eec6-fba8-4595-b28e-a7dba7512dbd None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:30.916484 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634bfe74-9695-47d7-895f-ebb29e748d29 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:30.954191 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d85b81a-8fca-49cd-8591-ed0534d2df82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:30.955174 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d85b81a-8fca-49cd-8591-ed0534d2df82 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:30.955738 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d85b81a-8fca-49cd-8591-ed0534d2df82 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:30.956230 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d85b81a-8fca-49cd-8591-ed0534d2df82 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:30.969939 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d85b81a-8fca-49cd-8591-ed0534d2df82 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:42:30.975686 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-2d85b81a-8fca-49cd-8591-ed0534d2df82 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:42:30.975686 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:30.975686 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:30.975686 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:30.975686 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:30.975686 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:30.975686 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:30.975686 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:30.975686 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:30.975686 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:30.975686 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:30.975686 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:30.975686 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:42:30.975686 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:42:30.975686 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:42:30.975686 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:42:30.976756 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:42:30.976756 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:42:30.976756 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:42:30.976756 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:42:30.976756 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:30.976756 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:30.976756 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:30.976756 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:30.976756 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:30.976756 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:30.976756 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:30.976756 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:30.976756 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:30.976756 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:30.976756 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:30.976756 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:30.977692 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:30.977692 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:30.977692 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:30.977692 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:30.977692 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:42:30.977692 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:42:30.977692 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:42:30.977692 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:42:30.977692 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:42:30.977692 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:42:30.977692 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:42:30.977692 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:30.980644 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 372/726] 10.222.0.52 () {56 vars in 1096 bytes} [Sun Sep 25 00:42:30 2022] GET /identity/v3/roles/admin => generated 83 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:30.990278 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d85b81a-8fca-49cd-8591-ed0534d2df82 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:31.009407 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:31.036963 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:31.088486 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6172eec6-fba8-4595-b28e-a7dba7512dbd None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:31.117369 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d85b81a-8fca-49cd-8591-ed0534d2df82 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:31.135134 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16305031-1e5b-499a-ace4-118b6aa21eb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:31.136023 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16305031-1e5b-499a-ace4-118b6aa21eb8 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:31.136304 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16305031-1e5b-499a-ace4-118b6aa21eb8 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:31.136573 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16305031-1e5b-499a-ace4-118b6aa21eb8 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:31.152933 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16305031-1e5b-499a-ace4-118b6aa21eb8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:42:31.157073 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1af866b-e01c-4919-9656-fb9a288bad1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:31.158006 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1af866b-e01c-4919-9656-fb9a288bad1c None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:31.158117 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-16305031-1e5b-499a-ace4-118b6aa21eb8 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 25 00:42:31.158117 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:31.158117 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:31.158117 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:31.158117 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:31.158117 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:31.158117 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:31.158117 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:31.158117 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:31.158117 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:31.158117 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:31.158117 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:31.158117 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:42:31.158117 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:42:31.158117 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:42:31.158117 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:42:31.159017 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:42:31.159017 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:42:31.159017 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:42:31.159017 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:42:31.159017 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:31.159017 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:31.159017 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:31.159017 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:31.159017 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:31.159017 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:31.159017 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:31.159017 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:31.159017 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:31.159017 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:31.159017 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:31.159017 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:31.159917 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:31.159917 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:31.159917 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:31.159917 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:31.159917 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:42:31.159917 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:42:31.159917 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:42:31.159917 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:42:31.159917 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:42:31.159917 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:42:31.159917 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 25 00:42:31.159917 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:31.161367 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1af866b-e01c-4919-9656-fb9a288bad1c None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:31.161846 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1af866b-e01c-4919-9656-fb9a288bad1c None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:31.162737 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1af866b-e01c-4919-9656-fb9a288bad1c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:31.163431 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1af866b-e01c-4919-9656-fb9a288bad1c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:31.164406 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 355/727] 10.222.0.52 () {56 vars in 1102 bytes} [Sun Sep 25 00:42:30 2022] GET /identity/v3/roles/service => generated 85 bytes in 250 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:31.165352 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16305031-1e5b-499a-ace4-118b6aa21eb8 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:31.166064 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1af866b-e01c-4919-9656-fb9a288bad1c None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:31.176199 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 373/728] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:42:30 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:31.177429 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1af866b-e01c-4919-9656-fb9a288bad1c None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:31.303812 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:31.316406 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1af866b-e01c-4919-9656-fb9a288bad1c None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:31.350386 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16305031-1e5b-499a-ace4-118b6aa21eb8 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:31.365292 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-957c325b-e49d-4371-9516-829a629020b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:31.365292 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957c325b-e49d-4371-9516-829a629020b5 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:31.365292 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957c325b-e49d-4371-9516-829a629020b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:31.365292 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957c325b-e49d-4371-9516-829a629020b5 None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:31.365921 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-957c325b-e49d-4371-9516-829a629020b5 None admin] RBAC: Adding query filter params (name=service) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:31.365921 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-957c325b-e49d-4371-9516-829a629020b5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:31.366255 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-957c325b-e49d-4371-9516-829a629020b5 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:31.375975 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:31.382603 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 374/729] 10.222.0.52 () {56 vars in 1103 bytes} [Sun Sep 25 00:42:31 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:31.383752 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-957c325b-e49d-4371-9516-829a629020b5 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:31.398297 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3111163c-4a35-429e-9468-3ccd947edb25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:31.399089 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3111163c-4a35-429e-9468-3ccd947edb25 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:31.399369 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3111163c-4a35-429e-9468-3ccd947edb25 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:31.399638 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3111163c-4a35-429e-9468-3ccd947edb25 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:31.411987 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3111163c-4a35-429e-9468-3ccd947edb25 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:42:31.417965 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-3111163c-4a35-429e-9468-3ccd947edb25 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:42:31.417965 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:31.417965 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:31.417965 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:31.417965 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:31.417965 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:31.417965 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:31.417965 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:31.417965 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:31.417965 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:31.417965 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:31.417965 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:31.417965 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:42:31.417965 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:42:31.417965 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:42:31.417965 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:42:31.419052 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:42:31.419052 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:42:31.419052 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:42:31.419052 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:42:31.419052 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:31.419052 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:31.419052 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:31.419052 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:31.419052 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:31.419052 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:31.419052 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:31.419052 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:31.419052 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:31.419052 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:31.419052 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:31.419052 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:31.420036 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:31.420036 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:31.420036 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:31.420036 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:31.420036 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:42:31.420036 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:42:31.420036 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:42:31.420036 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:42:31.420036 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:42:31.420036 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:42:31.420036 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:42:31.420036 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:31.424194 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 356/730] 10.222.0.52 () {56 vars in 1096 bytes} [Sun Sep 25 00:42:31 2022] GET /identity/v3/roles/admin => generated 83 bytes in 254 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:31.424991 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3111163c-4a35-429e-9468-3ccd947edb25 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:31.520420 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-957c325b-e49d-4371-9516-829a629020b5 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:31.540607 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3111163c-4a35-429e-9468-3ccd947edb25 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:31.557173 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69ccf8f-f1e2-4b35-ba74-d74fa810ae3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:31.557837 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69ccf8f-f1e2-4b35-ba74-d74fa810ae3e None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:31.558106 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69ccf8f-f1e2-4b35-ba74-d74fa810ae3e None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:31.558374 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69ccf8f-f1e2-4b35-ba74-d74fa810ae3e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:31.569821 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d69ccf8f-f1e2-4b35-ba74-d74fa810ae3e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:31.571276 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d69ccf8f-f1e2-4b35-ba74-d74fa810ae3e None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:31.577360 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe5c6e7-2e2c-4674-8f02-eea2430b7268 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:31.578436 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe5c6e7-2e2c-4674-8f02-eea2430b7268 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:31.578718 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe5c6e7-2e2c-4674-8f02-eea2430b7268 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:31.578987 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe5c6e7-2e2c-4674-8f02-eea2430b7268 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:31.586640 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-d69ccf8f-f1e2-4b35-ba74-d74fa810ae3e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:31.586640 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:31.586640 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:31.586640 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:31.586640 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:31.586640 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:31.586640 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:31.586640 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:31.586640 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:31.586640 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:31.586640 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:31.586640 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:31.586640 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:31.586640 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:31.586640 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:31.586640 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:31.588352 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:31.588352 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:31.588352 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:31.588352 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:31.588352 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:31.588352 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:31.588352 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:31.588352 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:31.588352 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:31.588352 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:31.588352 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:31.588352 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:31.588352 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:31.588352 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:31.588352 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:31.588352 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:31.588352 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:31.589384 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:31.589384 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:31.589384 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:31.592512 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 375/731] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:31.593376 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69ccf8f-f1e2-4b35-ba74-d74fa810ae3e None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:31.595764 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfe5c6e7-2e2c-4674-8f02-eea2430b7268 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:31.597921 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfe5c6e7-2e2c-4674-8f02-eea2430b7268 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:31.604124 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:31.614711 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-dfe5c6e7-2e2c-4674-8f02-eea2430b7268 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:31.614711 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:31.614711 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:31.614711 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:31.614711 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:31.614711 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:31.614711 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:31.614711 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:31.614711 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:31.614711 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:31.614711 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:31.614711 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:31.614711 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:31.614711 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:31.614711 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:31.614711 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:31.615739 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:31.615739 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:31.615739 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:31.615739 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:31.615739 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:31.615739 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:31.615739 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:31.615739 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:31.615739 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:31.615739 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:31.615739 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:31.615739 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:31.615739 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:31.615739 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:31.615739 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:31.615739 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:31.615739 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:31.617178 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:31.617178 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:31.617178 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:31.617178 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 357/732] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:31.617950 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8acc57b-070a-48e7-abb1-bba0fa813e4c None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:31.618227 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8acc57b-070a-48e7-abb1-bba0fa813e4c None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:31.618504 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8acc57b-070a-48e7-abb1-bba0fa813e4c None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:31.619871 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 358/733] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:42:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:42:31.624671 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8acc57b-070a-48e7-abb1-bba0fa813e4c None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:31.657986 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:31.716556 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69ccf8f-f1e2-4b35-ba74-d74fa810ae3e None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:31.741628 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8acc57b-070a-48e7-abb1-bba0fa813e4c None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:31.753203 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef3cb1e3-68af-40a6-92f0-4967e3f14f5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:31.753836 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3cb1e3-68af-40a6-92f0-4967e3f14f5f None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:31.754109 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3cb1e3-68af-40a6-92f0-4967e3f14f5f None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:31.754374 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3cb1e3-68af-40a6-92f0-4967e3f14f5f None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:31.755021 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef3cb1e3-68af-40a6-92f0-4967e3f14f5f None admin] RBAC: Adding query filter params (name=admin) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:31.755326 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef3cb1e3-68af-40a6-92f0-4967e3f14f5f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:31.756768 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef3cb1e3-68af-40a6-92f0-4967e3f14f5f None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:31.765703 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 376/734] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:42:31 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:31.766528 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef3cb1e3-68af-40a6-92f0-4967e3f14f5f None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:31.784383 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd2096ae-8cbe-4009-8d2b-efd250ddd601 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:31.785289 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2096ae-8cbe-4009-8d2b-efd250ddd601 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:31.785707 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2096ae-8cbe-4009-8d2b-efd250ddd601 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:31.786112 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2096ae-8cbe-4009-8d2b-efd250ddd601 None admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:31.786955 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd2096ae-8cbe-4009-8d2b-efd250ddd601 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:31.787252 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd2096ae-8cbe-4009-8d2b-efd250ddd601 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:31.788720 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd2096ae-8cbe-4009-8d2b-efd250ddd601 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:31.805898 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 359/735] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:31.805898 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f78da2-d6cf-4912-a589-51ec7ba410b3 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:31.805898 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f78da2-d6cf-4912-a589-51ec7ba410b3 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:31.805898 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f78da2-d6cf-4912-a589-51ec7ba410b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:31.822698 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-d0f78da2-d6cf-4912-a589-51ec7ba410b3 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:42:31.823895 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-d0f78da2-d6cf-4912-a589-51ec7ba410b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:42:31.848847 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:31.888928 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef3cb1e3-68af-40a6-92f0-4967e3f14f5f None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:31.902302 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 360/736] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:42:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:42:31.902302 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0f78da2-d6cf-4912-a589-51ec7ba410b3 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:31.929211 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:31.932407 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b5a38f-77b1-41a9-b18e-c2915d6136a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:31.933197 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b5a38f-77b1-41a9-b18e-c2915d6136a5 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:31.933638 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b5a38f-77b1-41a9-b18e-c2915d6136a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:31.934041 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b5a38f-77b1-41a9-b18e-c2915d6136a5 None admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:31.934843 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3b5a38f-77b1-41a9-b18e-c2915d6136a5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:31.935282 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3b5a38f-77b1-41a9-b18e-c2915d6136a5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:31.936778 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3b5a38f-77b1-41a9-b18e-c2915d6136a5 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:31.950833 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 377/737] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:31.952095 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b5a38f-77b1-41a9-b18e-c2915d6136a5 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:32.062688 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0f78da2-d6cf-4912-a589-51ec7ba410b3 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:32.082008 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b5a38f-77b1-41a9-b18e-c2915d6136a5 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:32.110048 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77bade08-104f-45c7-8ac1-be8357dde4f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:32.110982 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77bade08-104f-45c7-8ac1-be8357dde4f4 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:32.111399 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77bade08-104f-45c7-8ac1-be8357dde4f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:32.111837 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77bade08-104f-45c7-8ac1-be8357dde4f4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:32.117776 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:32.125834 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77bade08-104f-45c7-8ac1-be8357dde4f4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:32.127462 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab294d88-d0e6-4d56-8bfe-aab764c817a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:32.127641 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77bade08-104f-45c7-8ac1-be8357dde4f4 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:32.128101 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab294d88-d0e6-4d56-8bfe-aab764c817a8 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:32.128373 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab294d88-d0e6-4d56-8bfe-aab764c817a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:32.128653 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab294d88-d0e6-4d56-8bfe-aab764c817a8 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:32.138452 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab294d88-d0e6-4d56-8bfe-aab764c817a8 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:32.140426 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab294d88-d0e6-4d56-8bfe-aab764c817a8 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:32.146380 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-77bade08-104f-45c7-8ac1-be8357dde4f4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:32.146380 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:32.146380 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:32.146380 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:32.146380 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:32.146380 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:32.146380 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:32.146380 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:32.146380 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:32.146380 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:32.146380 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:32.146380 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:32.146380 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:32.146380 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:32.146380 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:32.146380 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:32.148231 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:32.148231 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:32.148231 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:32.148231 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:32.148231 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:32.148231 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:32.148231 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:32.148231 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:32.148231 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:32.148231 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:32.148231 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:32.148231 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:32.148231 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:32.148231 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:32.148231 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:32.148231 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:32.148231 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:32.149496 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:32.149496 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:32.149496 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:32.151392 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 361/738] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 254 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:32.152522 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77bade08-104f-45c7-8ac1-be8357dde4f4 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:32.159136 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-ab294d88-d0e6-4d56-8bfe-aab764c817a8 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 25 00:42:32.159136 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:32.159136 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:32.159136 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:32.159136 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:32.159136 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:32.159136 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:32.159136 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:32.159136 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:32.159136 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:32.159136 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:32.159136 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:32.159136 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 00:42:32.159136 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 00:42:32.159136 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 00:42:32.159136 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 00:42:32.160137 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:32.160137 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:32.160137 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 00:42:32.160137 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:32.160137 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 00:42:32.160137 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:32.160137 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:32.160137 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:32.160137 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:32.160137 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:32.160137 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:32.160137 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:32.160137 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:32.160137 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:32.160137 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:32.160137 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:32.160137 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:32.160137 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:32.161078 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:32.161078 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:32.161078 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 00:42:32.161078 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 00:42:32.161078 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 00:42:32.161078 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 00:42:32.161078 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 00:42:32.161078 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 00:42:32.161078 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 25 00:42:32.161078 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:32.162782 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 378/739] 10.222.0.52 () {56 vars in 1093 bytes} [Sun Sep 25 00:42:31 2022] GET /identity/v3/users/nova => generated 82 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:32.164368 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab294d88-d0e6-4d56-8bfe-aab764c817a8 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:32.240186 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:32.272887 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77bade08-104f-45c7-8ac1-be8357dde4f4 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:32.307853 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d9846a-cf30-4675-986f-27d7f9b7c0ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:32.308477 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d9846a-cf30-4675-986f-27d7f9b7c0ed None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:32.308750 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d9846a-cf30-4675-986f-27d7f9b7c0ed None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:32.309014 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d9846a-cf30-4675-986f-27d7f9b7c0ed None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:32.316661 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d9846a-cf30-4675-986f-27d7f9b7c0ed None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:32.318057 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d9846a-cf30-4675-986f-27d7f9b7c0ed None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:32.321937 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab294d88-d0e6-4d56-8bfe-aab764c817a8 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:32.330177 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-d1d9846a-cf30-4675-986f-27d7f9b7c0ed None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 25 00:42:32.330177 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:32.330177 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:32.330177 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:32.330177 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:32.330177 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:32.330177 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:32.330177 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:32.330177 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:32.330177 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:32.330177 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:32.330177 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:32.330177 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 00:42:32.330177 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 00:42:32.330177 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 00:42:32.330177 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 00:42:32.331279 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:32.331279 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:32.331279 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 00:42:32.331279 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:32.331279 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 00:42:32.331279 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:32.331279 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:32.331279 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:32.331279 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:32.331279 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:32.331279 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:32.331279 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:32.331279 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:32.331279 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:32.331279 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:32.331279 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:32.331279 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:32.331279 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:32.332413 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:32.332413 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:32.332413 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 00:42:32.332413 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 00:42:32.332413 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 00:42:32.332413 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 00:42:32.332413 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 00:42:32.332413 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 00:42:32.332413 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 25 00:42:32.332413 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:32.333394 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 362/740] 10.222.0.52 () {56 vars in 1117 bytes} [Sun Sep 25 00:42:32 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:32.334363 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d9846a-cf30-4675-986f-27d7f9b7c0ed None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:32.378029 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23cdb17a-ee53-4814-90c1-f2fedb34c639 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:32.378626 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23cdb17a-ee53-4814-90c1-f2fedb34c639 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:32.378909 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23cdb17a-ee53-4814-90c1-f2fedb34c639 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:32.379175 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23cdb17a-ee53-4814-90c1-f2fedb34c639 None admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:32.379811 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23cdb17a-ee53-4814-90c1-f2fedb34c639 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:32.380114 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23cdb17a-ee53-4814-90c1-f2fedb34c639 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:32.381501 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23cdb17a-ee53-4814-90c1-f2fedb34c639 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:32.393944 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 379/741] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:32.394752 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23cdb17a-ee53-4814-90c1-f2fedb34c639 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:32.444862 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d9846a-cf30-4675-986f-27d7f9b7c0ed None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:32.447421 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:32.474376 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:32.483520 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4674f7d7-3a31-421f-8e6b-d8b9aa359f39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:32.483520 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4674f7d7-3a31-421f-8e6b-d8b9aa359f39 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:32.483876 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4674f7d7-3a31-421f-8e6b-d8b9aa359f39 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:32.483876 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4674f7d7-3a31-421f-8e6b-d8b9aa359f39 None admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:32.484610 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4674f7d7-3a31-421f-8e6b-d8b9aa359f39 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:32.484908 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4674f7d7-3a31-421f-8e6b-d8b9aa359f39 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:32.486370 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4674f7d7-3a31-421f-8e6b-d8b9aa359f39 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:32.508550 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23cdb17a-ee53-4814-90c1-f2fedb34c639 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:32.508872 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 363/742] 10.222.0.52 () {56 vars in 1133 bytes} [Sun Sep 25 00:42:32 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:32.510172 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655442cc-8eda-4c8f-85c7-d984a9daf6b5 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:32.510454 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655442cc-8eda-4c8f-85c7-d984a9daf6b5 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:32.510723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655442cc-8eda-4c8f-85c7-d984a9daf6b5 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:32.511927 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 364/743] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:42:32 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:42:32.512596 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-655442cc-8eda-4c8f-85c7-d984a9daf6b5 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:32.547774 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d40930f2-9c4c-4d19-9672-1b38cce9663b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:32.548369 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40930f2-9c4c-4d19-9672-1b38cce9663b None admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:32.548640 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40930f2-9c4c-4d19-9672-1b38cce9663b None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:32.548904 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40930f2-9c4c-4d19-9672-1b38cce9663b None admin] PATH_INFO: `/v3/services` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:32.549582 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d40930f2-9c4c-4d19-9672-1b38cce9663b None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:32.550965 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d40930f2-9c4c-4d19-9672-1b38cce9663b None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:32.567426 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 380/744] 10.222.0.52 () {60 vars in 1141 bytes} [Sun Sep 25 00:42:32 2022] POST /identity/v3/services => generated 251 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:42:32.568243 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d40930f2-9c4c-4d19-9672-1b38cce9663b None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:32.636933 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-655442cc-8eda-4c8f-85c7-d984a9daf6b5 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:32.679271 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3146b193-17f9-483b-a0ee-3ac89235721a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:32.679894 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3146b193-17f9-483b-a0ee-3ac89235721a None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:32.680316 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3146b193-17f9-483b-a0ee-3ac89235721a None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:32.680430 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3146b193-17f9-483b-a0ee-3ac89235721a None admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:32.681164 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3146b193-17f9-483b-a0ee-3ac89235721a None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:32.682055 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3146b193-17f9-483b-a0ee-3ac89235721a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:32.682993 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3146b193-17f9-483b-a0ee-3ac89235721a None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:32.697074 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d40930f2-9c4c-4d19-9672-1b38cce9663b None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:32.709595 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 365/745] 10.222.0.52 () {56 vars in 1149 bytes} [Sun Sep 25 00:42:32 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:32.709595 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3146b193-17f9-483b-a0ee-3ac89235721a None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:32.711775 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:32.743798 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5845a86-b8fb-4e5a-8b54-e2fa517fd3bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:32.744709 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5845a86-b8fb-4e5a-8b54-e2fa517fd3bd None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:32.744985 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5845a86-b8fb-4e5a-8b54-e2fa517fd3bd None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:32.745279 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5845a86-b8fb-4e5a-8b54-e2fa517fd3bd None admin] PATH_INFO: `/v3/users/swift` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:32.753766 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5845a86-b8fb-4e5a-8b54-e2fa517fd3bd None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:32.755216 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5845a86-b8fb-4e5a-8b54-e2fa517fd3bd None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:32.763146 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:32.768965 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-c5845a86-b8fb-4e5a-8b54-e2fa517fd3bd None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 25 00:42:32.768965 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:32.768965 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:32.768965 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:32.768965 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:32.768965 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:32.768965 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:32.768965 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:32.768965 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:32.768965 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:32.768965 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:32.768965 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:32.768965 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 00:42:32.768965 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 00:42:32.768965 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 00:42:32.768965 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 00:42:32.770200 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:32.770200 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:32.770200 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 00:42:32.770200 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:32.770200 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 00:42:32.770200 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:32.770200 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:32.770200 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:32.770200 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:32.770200 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:32.770200 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:32.770200 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:32.770200 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:32.770200 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:32.770200 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:32.770200 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:32.770200 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:32.770200 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:32.771437 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:32.771437 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:32.771437 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 00:42:32.771437 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 00:42:32.771437 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 00:42:32.771437 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 00:42:32.771437 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 00:42:32.771437 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 00:42:32.771437 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 25 00:42:32.771437 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:32.771437 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 381/746] 10.222.0.52 () {56 vars in 1096 bytes} [Sun Sep 25 00:42:32 2022] GET /identity/v3/users/swift => generated 83 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:32.771437 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38523822-e56f-427a-b759-f8f5650832f4 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:32.772315 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38523822-e56f-427a-b759-f8f5650832f4 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:32.772315 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38523822-e56f-427a-b759-f8f5650832f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:32.793819 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-38523822-e56f-427a-b759-f8f5650832f4 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:42:32.795017 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-38523822-e56f-427a-b759-f8f5650832f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:42:32.837535 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3146b193-17f9-483b-a0ee-3ac89235721a None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:32.872880 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 382/747] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:42:32 2022] POST /identity/v3/auth/tokens => generated 903 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 00:42:32.874157 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38523822-e56f-427a-b759-f8f5650832f4 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:32.876462 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d00feead-4ed6-4ec7-ab2c-85631eb9b488 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:32.877093 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00feead-4ed6-4ec7-ab2c-85631eb9b488 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:32.877394 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00feead-4ed6-4ec7-ab2c-85631eb9b488 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:32.877658 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00feead-4ed6-4ec7-ab2c-85631eb9b488 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:32.896827 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d00feead-4ed6-4ec7-ab2c-85631eb9b488 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:32.899247 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d00feead-4ed6-4ec7-ab2c-85631eb9b488 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:32.919184 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-d00feead-4ed6-4ec7-ab2c-85631eb9b488 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:32.919184 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:32.919184 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:32.919184 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:32.919184 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:32.919184 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:32.919184 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:32.919184 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:32.919184 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:32.919184 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:32.919184 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:32.919184 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:32.919184 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:32.919184 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:32.919184 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:32.919184 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:32.920225 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:32.920225 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:32.920225 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:32.920225 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:32.920225 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:32.920225 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:32.920225 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:32.920225 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:32.920225 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:32.920225 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:32.920225 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:32.920225 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:32.920225 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:32.920225 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:32.920225 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:32.920225 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:32.920225 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:32.921157 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:32.921157 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:32.921157 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:32.923379 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 366/748] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:32.924380 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d00feead-4ed6-4ec7-ab2c-85631eb9b488 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:32.992836 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:32.997148 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38523822-e56f-427a-b759-f8f5650832f4 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:33.037050 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f48e8ac1-71c2-46ea-a1e8-182b7b77b0f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:33.038096 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48e8ac1-71c2-46ea-a1e8-182b7b77b0f3 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:33.038586 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48e8ac1-71c2-46ea-a1e8-182b7b77b0f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:33.038975 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48e8ac1-71c2-46ea-a1e8-182b7b77b0f3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:33.066149 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:33.077827 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d00feead-4ed6-4ec7-ab2c-85631eb9b488 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:33.122322 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abbff0c8-6e99-41e4-918c-185aeafc42db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:33.122845 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbff0c8-6e99-41e4-918c-185aeafc42db None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:33.123119 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbff0c8-6e99-41e4-918c-185aeafc42db None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:33.123389 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbff0c8-6e99-41e4-918c-185aeafc42db None admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:33.124176 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abbff0c8-6e99-41e4-918c-185aeafc42db None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:33.124478 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abbff0c8-6e99-41e4-918c-185aeafc42db None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:33.126166 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abbff0c8-6e99-41e4-918c-185aeafc42db None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:33.129209 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f48e8ac1-71c2-46ea-a1e8-182b7b77b0f3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:33.131798 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f48e8ac1-71c2-46ea-a1e8-182b7b77b0f3 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:33.150166 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 367/749] 10.222.0.52 () {56 vars in 1135 bytes} [Sun Sep 25 00:42:32 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:33.151064 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abbff0c8-6e99-41e4-918c-185aeafc42db None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:33.250669 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:33.258398 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 383/750] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:42:32 2022] GET /identity/v3/auth/tokens => generated 903 bytes in 384 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 00:42:33.259745 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f48e8ac1-71c2-46ea-a1e8-182b7b77b0f3 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:33.276592 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abbff0c8-6e99-41e4-918c-185aeafc42db None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:33.323997 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf378f2-1458-4546-9283-301e77e7b734 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:33.324510 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf378f2-1458-4546-9283-301e77e7b734 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:33.324785 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf378f2-1458-4546-9283-301e77e7b734 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:33.325050 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf378f2-1458-4546-9283-301e77e7b734 None admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:33.325733 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf378f2-1458-4546-9283-301e77e7b734 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:33.326178 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf378f2-1458-4546-9283-301e77e7b734 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:33.327987 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf378f2-1458-4546-9283-301e77e7b734 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:33.347870 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 368/751] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:33.348592 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf378f2-1458-4546-9283-301e77e7b734 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:33.368574 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:33.391791 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f48e8ac1-71c2-46ea-a1e8-182b7b77b0f3 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:33.438562 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34855cf0-c2c3-4dfd-8d78-564a059e4e4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:33.439297 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34855cf0-c2c3-4dfd-8d78-564a059e4e4c None admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:33.439602 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34855cf0-c2c3-4dfd-8d78-564a059e4e4c None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:33.440046 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34855cf0-c2c3-4dfd-8d78-564a059e4e4c None admin] PATH_INFO: `/v3/services` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:33.440751 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34855cf0-c2c3-4dfd-8d78-564a059e4e4c None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:33.442786 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34855cf0-c2c3-4dfd-8d78-564a059e4e4c None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:33.465943 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 384/752] 10.222.0.52 () {60 vars in 1141 bytes} [Sun Sep 25 00:42:33 2022] POST /identity/v3/services => generated 240 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:42:33.465943 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34855cf0-c2c3-4dfd-8d78-564a059e4e4c None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:33.515386 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf378f2-1458-4546-9283-301e77e7b734 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:33.567565 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:33.574895 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b6c954-6c1a-4e4c-a634-1c821907a290 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:33.575686 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b6c954-6c1a-4e4c-a634-1c821907a290 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:33.576080 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b6c954-6c1a-4e4c-a634-1c821907a290 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:33.576450 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b6c954-6c1a-4e4c-a634-1c821907a290 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:33.647160 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34855cf0-c2c3-4dfd-8d78-564a059e4e4c None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:33.664766 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5b6c954-6c1a-4e4c-a634-1c821907a290 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:33.666411 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5b6c954-6c1a-4e4c-a634-1c821907a290 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:33.678061 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:33.700394 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda43395-8c07-481b-8eb2-ac2f828c58d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:33.701396 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda43395-8c07-481b-8eb2-ac2f828c58d6 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:33.701690 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda43395-8c07-481b-8eb2-ac2f828c58d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:33.701943 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda43395-8c07-481b-8eb2-ac2f828c58d6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:33.714721 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda43395-8c07-481b-8eb2-ac2f828c58d6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:33.716887 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda43395-8c07-481b-8eb2-ac2f828c58d6 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:33.733883 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-fda43395-8c07-481b-8eb2-ac2f828c58d6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:33.733883 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:33.733883 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:33.733883 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:33.733883 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:33.733883 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:33.733883 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:33.733883 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:33.733883 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:33.733883 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:33.733883 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:33.733883 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:33.733883 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:33.733883 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:33.733883 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:33.733883 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:33.735054 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:33.735054 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:33.735054 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:33.735054 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:33.735054 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:33.735054 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:33.735054 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:33.735054 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:33.735054 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:33.735054 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:33.735054 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:33.735054 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:33.735054 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:33.735054 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:33.735054 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:33.735054 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:33.735054 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:33.736145 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:33.736145 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:33.736145 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:33.738699 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 385/753] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 275 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:33.739706 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda43395-8c07-481b-8eb2-ac2f828c58d6 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:33.806413 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 369/754] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:42:33 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 458 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 25 00:42:33.807294 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b6c954-6c1a-4e4c-a634-1c821907a290 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:33.865007 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda43395-8c07-481b-8eb2-ac2f828c58d6 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:33.880299 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:33.911500 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7228b827-8681-4906-b6f8-4321d4600846 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:33.912154 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7228b827-8681-4906-b6f8-4321d4600846 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:33.913745 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7228b827-8681-4906-b6f8-4321d4600846 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:33.913745 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7228b827-8681-4906-b6f8-4321d4600846 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:33.921723 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7228b827-8681-4906-b6f8-4321d4600846 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:33.923401 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7228b827-8681-4906-b6f8-4321d4600846 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:33.938065 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-7228b827-8681-4906-b6f8-4321d4600846 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:42:33.938065 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:33.938065 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:33.938065 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:33.938065 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:33.938065 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:33.938065 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:33.938065 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:33.938065 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:33.938065 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:33.938065 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:33.938065 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:33.938065 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 00:42:33.938065 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 00:42:33.938065 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 00:42:33.939267 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 00:42:33.939267 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:33.939267 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:33.939267 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:33.939267 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:33.939267 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:33.939267 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:33.939267 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:33.939267 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:33.939267 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:33.939267 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:33.939267 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:33.939267 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:33.939267 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:33.939267 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:33.939267 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:33.939267 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:33.941087 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 00:42:33.941087 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 00:42:33.941087 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 00:42:33.941087 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 00:42:33.941087 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 00:42:33.941087 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 00:42:33.941087 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:42:33.941087 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:33.942122 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 386/755] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:42:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:33.945558 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7228b827-8681-4906-b6f8-4321d4600846 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:33.958260 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b6c954-6c1a-4e4c-a634-1c821907a290 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:33.969676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:34.000833 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d65a882b-d8e2-4151-9807-21997f2a9c40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:34.001494 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65a882b-d8e2-4151-9807-21997f2a9c40 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:34.001885 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65a882b-d8e2-4151-9807-21997f2a9c40 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:34.002325 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65a882b-d8e2-4151-9807-21997f2a9c40 None admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:34.002959 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d65a882b-d8e2-4151-9807-21997f2a9c40 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:34.003330 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d65a882b-d8e2-4151-9807-21997f2a9c40 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:34.005036 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d65a882b-d8e2-4151-9807-21997f2a9c40 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:34.024138 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 370/756] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:34.025141 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d65a882b-d8e2-4151-9807-21997f2a9c40 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:34.081859 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7228b827-8681-4906-b6f8-4321d4600846 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:34.131903 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab672731-f93d-47a7-b1de-522368ef39ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:34.132950 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab672731-f93d-47a7-b1de-522368ef39ca None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:34.133438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab672731-f93d-47a7-b1de-522368ef39ca None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:34.133900 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab672731-f93d-47a7-b1de-522368ef39ca None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:34.148625 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab672731-f93d-47a7-b1de-522368ef39ca None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:34.150580 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab672731-f93d-47a7-b1de-522368ef39ca None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:34.153647 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d65a882b-d8e2-4151-9807-21997f2a9c40 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:34.169750 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-ab672731-f93d-47a7-b1de-522368ef39ca None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:34.169750 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:34.169750 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:34.169750 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:34.169750 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:34.169750 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:34.169750 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:34.169750 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:34.169750 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:34.169750 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:34.169750 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:34.169750 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:34.169750 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:34.169750 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:34.169750 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:34.169750 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:34.170732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:34.170732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:34.170732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:34.170732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:34.170732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:34.170732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:34.170732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:34.170732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:34.170732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:34.170732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:34.170732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:34.170732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:34.170732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:34.170732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:34.170732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:34.170732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:34.170732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:34.171630 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:34.171630 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:34.171630 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:34.173880 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 387/757] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:34.175322 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab672731-f93d-47a7-b1de-522368ef39ca None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:34.193348 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2383e074-8520-4e0d-b8f0-1d003f985a3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:34.193954 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2383e074-8520-4e0d-b8f0-1d003f985a3d None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:34.194763 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2383e074-8520-4e0d-b8f0-1d003f985a3d None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:34.195207 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2383e074-8520-4e0d-b8f0-1d003f985a3d None admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:34.196045 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2383e074-8520-4e0d-b8f0-1d003f985a3d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:34.196346 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2383e074-8520-4e0d-b8f0-1d003f985a3d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:34.198215 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2383e074-8520-4e0d-b8f0-1d003f985a3d None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:34.209338 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:34.218316 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 371/758] 10.222.0.52 () {56 vars in 1148 bytes} [Sun Sep 25 00:42:34 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:34.218316 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2383e074-8520-4e0d-b8f0-1d003f985a3d None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:34.249666 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:34.309678 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab672731-f93d-47a7-b1de-522368ef39ca None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:34.342214 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2383e074-8520-4e0d-b8f0-1d003f985a3d None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:34.350164 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb0ea2f-917b-4e2e-bf95-7ee0b609e882 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:34.351503 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb0ea2f-917b-4e2e-bf95-7ee0b609e882 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:34.351961 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb0ea2f-917b-4e2e-bf95-7ee0b609e882 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:34.352235 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb0ea2f-917b-4e2e-bf95-7ee0b609e882 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:34.359502 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfb0ea2f-917b-4e2e-bf95-7ee0b609e882 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:34.361108 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfb0ea2f-917b-4e2e-bf95-7ee0b609e882 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:34.375683 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-dfb0ea2f-917b-4e2e-bf95-7ee0b609e882 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:42:34.375683 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:34.375683 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:34.375683 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:34.375683 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:34.375683 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:34.375683 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:34.375683 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:34.375683 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:34.375683 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:34.375683 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:34.375683 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:34.375683 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 00:42:34.375683 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 00:42:34.375683 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 00:42:34.376651 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 00:42:34.376651 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:34.376651 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:34.376651 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:34.376651 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:34.376651 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:34.376651 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:34.376651 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:34.376651 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:34.376651 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:34.376651 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:34.376651 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:34.376651 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:34.376651 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:34.376651 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:34.376651 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:34.376651 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:34.377594 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 00:42:34.377594 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 00:42:34.377594 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 00:42:34.377594 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 00:42:34.377594 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 00:42:34.377594 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 00:42:34.377594 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:42:34.377594 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:34.377594 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 388/759] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:42:34 2022] GET /identity/v3/projects/service => generated 88 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:34.378618 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb0ea2f-917b-4e2e-bf95-7ee0b609e882 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:34.393227 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b07842a-220f-49ac-bc75-3355aee67127 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:34.395041 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b07842a-220f-49ac-bc75-3355aee67127 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:34.395303 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b07842a-220f-49ac-bc75-3355aee67127 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:34.395571 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b07842a-220f-49ac-bc75-3355aee67127 None admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:34.396386 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b07842a-220f-49ac-bc75-3355aee67127 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:34.396682 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b07842a-220f-49ac-bc75-3355aee67127 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:34.398613 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b07842a-220f-49ac-bc75-3355aee67127 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:34.424798 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 372/760] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:34.425863 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b07842a-220f-49ac-bc75-3355aee67127 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:34.501237 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:34.530658 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb0ea2f-917b-4e2e-bf95-7ee0b609e882 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:34.570908 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:34.590827 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b07842a-220f-49ac-bc75-3355aee67127 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:34.591026 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007a4a4d-43d6-4183-a8a8-e97416bd33b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:34.591600 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007a4a4d-43d6-4183-a8a8-e97416bd33b4 None admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:34.591871 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007a4a4d-43d6-4183-a8a8-e97416bd33b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:34.592134 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007a4a4d-43d6-4183-a8a8-e97416bd33b4 None admin] PATH_INFO: `/v3/projects/9db0fd07fc494203b2ada13f3cd046d3/users/f142988a4dd14a5f96eacea45c046f85/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:34.594218 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007a4a4d-43d6-4183-a8a8-e97416bd33b4 None admin] RBAC: Authorizing `identity:create_grant(project_id=9db0fd07fc494203b2ada13f3cd046d3, user_id=f142988a4dd14a5f96eacea45c046f85, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:34.595827 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007a4a4d-43d6-4183-a8a8-e97416bd33b4 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:34.620818 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 389/761] 10.222.0.52 () {58 vars in 1439 bytes} [Sun Sep 25 00:42:34 2022] PUT /identity/v3/projects/9db0fd07fc494203b2ada13f3cd046d3/users/f142988a4dd14a5f96eacea45c046f85/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:42:34.620818 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007a4a4d-43d6-4183-a8a8-e97416bd33b4 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:34.679722 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa8c45c-020e-4e71-9c4b-6258d6e4641b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:34.680262 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa8c45c-020e-4e71-9c4b-6258d6e4641b None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:34.680537 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa8c45c-020e-4e71-9c4b-6258d6e4641b None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:34.680804 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa8c45c-020e-4e71-9c4b-6258d6e4641b None admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:34.681539 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baa8c45c-020e-4e71-9c4b-6258d6e4641b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:34.681842 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baa8c45c-020e-4e71-9c4b-6258d6e4641b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:34.683621 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baa8c45c-020e-4e71-9c4b-6258d6e4641b None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:34.728049 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 373/762] 10.222.0.52 () {56 vars in 1148 bytes} [Sun Sep 25 00:42:34 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 303 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:34.734448 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa8c45c-020e-4e71-9c4b-6258d6e4641b None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:34.765933 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007a4a4d-43d6-4183-a8a8-e97416bd33b4 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:34.845213 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7099bd7-e3b6-4a34-95c5-0b1a13d2906a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:34.845884 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7099bd7-e3b6-4a34-95c5-0b1a13d2906a None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:34.846159 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7099bd7-e3b6-4a34-95c5-0b1a13d2906a None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:34.846421 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7099bd7-e3b6-4a34-95c5-0b1a13d2906a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:34.857674 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7099bd7-e3b6-4a34-95c5-0b1a13d2906a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:34.860606 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7099bd7-e3b6-4a34-95c5-0b1a13d2906a None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:34.878803 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-a7099bd7-e3b6-4a34-95c5-0b1a13d2906a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:42:34.878803 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:34.878803 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:34.878803 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:34.878803 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:34.878803 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:34.878803 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:34.878803 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:34.878803 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:34.878803 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:34.878803 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:34.878803 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:34.878803 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 00:42:34.878803 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 00:42:34.878803 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 00:42:34.879821 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 00:42:34.879821 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:34.879821 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:34.879821 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:34.879821 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:34.879821 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:34.879821 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:34.879821 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:34.879821 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:34.879821 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:34.879821 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:34.879821 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:34.879821 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:34.879821 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:34.879821 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:34.879821 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:34.879821 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:34.880717 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 00:42:34.880717 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 00:42:34.880717 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 00:42:34.880717 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 00:42:34.880717 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 00:42:34.880717 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 00:42:34.880717 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:42:34.880717 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:34.880717 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 390/763] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:42:34 2022] GET /identity/v3/projects/service => generated 88 bytes in 266 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:34.885830 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7099bd7-e3b6-4a34-95c5-0b1a13d2906a None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:34.889064 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:34.962286 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa8c45c-020e-4e71-9c4b-6258d6e4641b None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:35.015059 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf047e57-cf45-47e0-bfce-f44199b82198 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:35.016191 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf047e57-cf45-47e0-bfce-f44199b82198 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:35.016672 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf047e57-cf45-47e0-bfce-f44199b82198 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:35.017167 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf047e57-cf45-47e0-bfce-f44199b82198 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:35.018403 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf047e57-cf45-47e0-bfce-f44199b82198 None admin] RBAC: Adding query filter params (role.id=a8e34e460e8a4ed7bcaa54da3c2d5a0c, scope.project.id=9db0fd07fc494203b2ada13f3cd046d3, user.id=3f20ffd8df994c1db1b1e7248db5b5a3) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:35.018934 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf047e57-cf45-47e0-bfce-f44199b82198 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:35.020752 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf047e57-cf45-47e0-bfce-f44199b82198 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:35.028094 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 374/764] 10.222.0.52 () {56 vars in 1374 bytes} [Sun Sep 25 00:42:34 2022] GET /identity/v3/role_assignments?user.id=3f20ffd8df994c1db1b1e7248db5b5a3&scope.project.id=9db0fd07fc494203b2ada13f3cd046d3&role.id=a8e34e460e8a4ed7bcaa54da3c2d5a0c => generated 259 bytes in 294 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:35.029873 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab1527a-3d8f-4af2-9292-328a7eb369e5 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:35.030380 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab1527a-3d8f-4af2-9292-328a7eb369e5 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:35.030877 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab1527a-3d8f-4af2-9292-328a7eb369e5 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:35.032920 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 375/765] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:42:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:42:35.046862 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a04e962-fa21-4985-bde3-454fc3133a53 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:35.047634 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a04e962-fa21-4985-bde3-454fc3133a53 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:35.048118 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a04e962-fa21-4985-bde3-454fc3133a53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:35.065786 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7099bd7-e3b6-4a34-95c5-0b1a13d2906a None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:35.103160 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-7a04e962-fa21-4985-bde3-454fc3133a53 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:42:35.106300 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-7a04e962-fa21-4985-bde3-454fc3133a53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:42:35.126082 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:35.167997 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79067de-f03f-40a0-8554-c4d690fccdd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:35.169480 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79067de-f03f-40a0-8554-c4d690fccdd8 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:35.170405 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79067de-f03f-40a0-8554-c4d690fccdd8 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:35.171175 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79067de-f03f-40a0-8554-c4d690fccdd8 None admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:35.173192 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e79067de-f03f-40a0-8554-c4d690fccdd8 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:35.174131 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e79067de-f03f-40a0-8554-c4d690fccdd8 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:35.177450 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e79067de-f03f-40a0-8554-c4d690fccdd8 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:35.204755 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 391/766] 10.222.0.52 () {56 vars in 1148 bytes} [Sun Sep 25 00:42:34 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 319 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:42:35.211242 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79067de-f03f-40a0-8554-c4d690fccdd8 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:35.282657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 376/767] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:42:35 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 237 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:42:35.351751 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:35.449994 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79067de-f03f-40a0-8554-c4d690fccdd8 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:35.539887 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f6fe20-7faa-4318-bdaf-09e0ce25e60a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:35.540745 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f6fe20-7faa-4318-bdaf-09e0ce25e60a None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:35.541184 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f6fe20-7faa-4318-bdaf-09e0ce25e60a None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:35.541662 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f6fe20-7faa-4318-bdaf-09e0ce25e60a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:35.542945 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6f6fe20-7faa-4318-bdaf-09e0ce25e60a None admin] RBAC: Adding query filter params (role.id=2c242abf6ed8410ab0a4a354ca1794fb, scope.project.id=9db0fd07fc494203b2ada13f3cd046d3, user.id=da606b3906dd4171bd76fd281e03e258) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:35.543464 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6f6fe20-7faa-4318-bdaf-09e0ce25e60a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:35.545972 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6f6fe20-7faa-4318-bdaf-09e0ce25e60a None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:35.566012 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 392/768] 10.222.0.52 () {56 vars in 1374 bytes} [Sun Sep 25 00:42:35 2022] GET /identity/v3/role_assignments?user.id=da606b3906dd4171bd76fd281e03e258&scope.project.id=9db0fd07fc494203b2ada13f3cd046d3&role.id=2c242abf6ed8410ab0a4a354ca1794fb => generated 259 bytes in 356 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:42:35.608731 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a04e962-fa21-4985-bde3-454fc3133a53 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:35.867971 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a04e962-fa21-4985-bde3-454fc3133a53 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:35.934641 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d04074-74ce-467e-879f-fb689185cce4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:35.935745 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d04074-74ce-467e-879f-fb689185cce4 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:35.936191 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d04074-74ce-467e-879f-fb689185cce4 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:35.936741 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d04074-74ce-467e-879f-fb689185cce4 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:35.937838 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d04074-74ce-467e-879f-fb689185cce4 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:35.940385 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d04074-74ce-467e-879f-fb689185cce4 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:35.959111 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-87d04074-74ce-467e-879f-fb689185cce4 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 25 00:42:35.959111 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:35.959111 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 00:42:35.959111 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 00:42:35.959111 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 00:42:35.959111 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 00:42:35.959111 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 00:42:35.959111 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:42:35.959111 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 25 00:42:35.959111 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:35.959111 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 00:42:35.959111 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:35.959111 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:35.959111 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:35.959111 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:35.959111 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:35.959111 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:35.963024 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:35.963024 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:35.963024 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:35.963024 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:35.963024 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:35.963024 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:35.963024 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 00:42:35.963024 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 00:42:35.963024 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 00:42:35.963024 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 00:42:35.963024 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:35.963024 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:35.963024 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:35.963024 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:35.963024 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:35.963024 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:35.966278 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:35.966278 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:35.966278 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:35.966278 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:35.966278 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:35.966278 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:35.966278 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:35.966278 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:35.966278 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:35.966278 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:35.966278 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 00:42:35.966278 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:42:35.966278 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 25 00:42:35.966278 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:35.966278 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 377/769] 10.222.0.52 () {56 vars in 1129 bytes} [Sun Sep 25 00:42:35 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 354 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:35.976243 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f6fe20-7faa-4318-bdaf-09e0ce25e60a None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:35.976567 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ecde1e-3ddc-4126-8ddd-5ddb6913bacd None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:35.977309 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ecde1e-3ddc-4126-8ddd-5ddb6913bacd None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:35.977933 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ecde1e-3ddc-4126-8ddd-5ddb6913bacd None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:35.980886 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 378/770] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:42:35 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:42:35.990564 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30f2aa1-7a43-4d4e-b234-69422a101c94 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:35.991259 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30f2aa1-7a43-4d4e-b234-69422a101c94 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:35.992100 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30f2aa1-7a43-4d4e-b234-69422a101c94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:36.015787 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:36.027197 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-d30f2aa1-7a43-4d4e-b234-69422a101c94 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:42:36.029170 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-d30f2aa1-7a43-4d4e-b234-69422a101c94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:42:36.224854 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f6fe20-7faa-4318-bdaf-09e0ce25e60a None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:36.235981 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 379/771] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:42:35 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 247 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 25 00:42:36.295632 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:36.313480 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-975747d8-b99c-4b8c-8d09-ae12a8d81020 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:36.314538 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975747d8-b99c-4b8c-8d09-ae12a8d81020 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:36.314996 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975747d8-b99c-4b8c-8d09-ae12a8d81020 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:36.315450 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975747d8-b99c-4b8c-8d09-ae12a8d81020 None admin] PATH_INFO: `/v3/services` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:36.316504 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-975747d8-b99c-4b8c-8d09-ae12a8d81020 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:36.316976 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-975747d8-b99c-4b8c-8d09-ae12a8d81020 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:36.319572 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-975747d8-b99c-4b8c-8d09-ae12a8d81020 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:36.328510 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 393/772] 10.222.0.52 () {56 vars in 1124 bytes} [Sun Sep 25 00:42:35 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 361 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:36.335881 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30f2aa1-7a43-4d4e-b234-69422a101c94 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:36.558165 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30f2aa1-7a43-4d4e-b234-69422a101c94 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:36.608257 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-975747d8-b99c-4b8c-8d09-ae12a8d81020 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:36.616645 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ab0d5b-419b-41c3-b1f5-ffb93dd5c4c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:36.617493 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ab0d5b-419b-41c3-b1f5-ffb93dd5c4c3 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:36.617773 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ab0d5b-419b-41c3-b1f5-ffb93dd5c4c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:36.618037 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ab0d5b-419b-41c3-b1f5-ffb93dd5c4c3 None admin] PATH_INFO: `/v3/services` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:36.618682 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75ab0d5b-419b-41c3-b1f5-ffb93dd5c4c3 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:36.618978 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75ab0d5b-419b-41c3-b1f5-ffb93dd5c4c3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:36.620387 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75ab0d5b-419b-41c3-b1f5-ffb93dd5c4c3 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:36.622979 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:36.628705 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 380/773] 10.222.0.52 () {56 vars in 1124 bytes} [Sun Sep 25 00:42:36 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 295 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:36.635429 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ab0d5b-419b-41c3-b1f5-ffb93dd5c4c3 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:36.742855 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-975747d8-b99c-4b8c-8d09-ae12a8d81020 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:36.785764 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f48b86a6-c0d4-4f7c-9325-05b7ebf46c72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:36.786407 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48b86a6-c0d4-4f7c-9325-05b7ebf46c72 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:36.786710 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48b86a6-c0d4-4f7c-9325-05b7ebf46c72 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:36.787093 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48b86a6-c0d4-4f7c-9325-05b7ebf46c72 None admin] PATH_INFO: `/v3/services/network` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:36.787714 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f48b86a6-c0d4-4f7c-9325-05b7ebf46c72 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:36.789462 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f48b86a6-c0d4-4f7c-9325-05b7ebf46c72 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:36.800166 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-f48b86a6-c0d4-4f7c-9325-05b7ebf46c72 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 25 00:42:36.800166 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:36.800166 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 00:42:36.800166 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 00:42:36.800166 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 00:42:36.800166 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 00:42:36.800166 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 00:42:36.800166 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:42:36.800166 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 25 00:42:36.800166 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:36.800166 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 00:42:36.800166 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:36.800166 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:36.800166 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:36.800166 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:36.800166 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:36.800166 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:36.801281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:36.801281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:36.801281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:36.801281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:36.801281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:36.801281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:36.801281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 00:42:36.801281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 00:42:36.801281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 00:42:36.801281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 00:42:36.801281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:36.801281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:36.801281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:36.801281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:36.801281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:36.801281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:36.802157 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:36.802157 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:36.802157 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:36.802157 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:36.802157 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:36.802157 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:36.802157 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:36.802157 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:36.802157 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:36.802157 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:36.802157 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 00:42:36.802157 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:42:36.802157 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 25 00:42:36.802157 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:36.802157 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 394/774] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:42:36 2022] GET /identity/v3/services/network => generated 88 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:42:36.808260 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f48b86a6-c0d4-4f7c-9325-05b7ebf46c72 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:36.875554 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:36.928894 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ab0d5b-419b-41c3-b1f5-ffb93dd5c4c3 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:36.937312 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f48b86a6-c0d4-4f7c-9325-05b7ebf46c72 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:36.977461 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f114f7-f0c7-4b65-86fb-4000ff926f4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:36.978092 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f114f7-f0c7-4b65-86fb-4000ff926f4d None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:36.978369 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f114f7-f0c7-4b65-86fb-4000ff926f4d None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:36.978640 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f114f7-f0c7-4b65-86fb-4000ff926f4d None admin] PATH_INFO: `/v3/services` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:36.979290 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83f114f7-f0c7-4b65-86fb-4000ff926f4d None admin] RBAC: Adding query filter params (name=network) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:36.979591 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83f114f7-f0c7-4b65-86fb-4000ff926f4d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:36.981005 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83f114f7-f0c7-4b65-86fb-4000ff926f4d None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:37.003176 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 395/775] 10.222.0.52 () {56 vars in 1112 bytes} [Sun Sep 25 00:42:36 2022] GET /identity/v3/services?name=network => generated 124 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:37.004140 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f114f7-f0c7-4b65-86fb-4000ff926f4d None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:37.025967 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-584f216c-f947-445f-a257-559c8203ae69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:37.026594 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584f216c-f947-445f-a257-559c8203ae69 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:37.026868 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584f216c-f947-445f-a257-559c8203ae69 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:37.027137 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584f216c-f947-445f-a257-559c8203ae69 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:37.027854 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-584f216c-f947-445f-a257-559c8203ae69 None admin] RBAC: Adding query filter params (interface=public, service_id=fb45b126daa042fb92bc07cdb4b9abc5, region_id=RegionOne) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:37.028162 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-584f216c-f947-445f-a257-559c8203ae69 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:37.029602 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-584f216c-f947-445f-a257-559c8203ae69 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:37.058248 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 381/776] 10.222.0.52 () {56 vars in 1251 bytes} [Sun Sep 25 00:42:36 2022] GET /identity/v3/endpoints?service_id=fb45b126daa042fb92bc07cdb4b9abc5&interface=public®ion_id=RegionOne => generated 194 bytes in 423 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:42:37.070483 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-584f216c-f947-445f-a257-559c8203ae69 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:37.182721 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f114f7-f0c7-4b65-86fb-4000ff926f4d None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:37.217152 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:37.218451 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:37.252509 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a0aefa7-ec74-493e-9dae-71af98ec3f7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:37.253564 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0aefa7-ec74-493e-9dae-71af98ec3f7e None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:37.253958 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0aefa7-ec74-493e-9dae-71af98ec3f7e None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:37.254389 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0aefa7-ec74-493e-9dae-71af98ec3f7e None admin] PATH_INFO: `/v3/services` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:37.255434 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a0aefa7-ec74-493e-9dae-71af98ec3f7e None admin] RBAC: Adding query filter params (type=network) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:37.255920 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a0aefa7-ec74-493e-9dae-71af98ec3f7e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:37.259798 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a0aefa7-ec74-493e-9dae-71af98ec3f7e None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:37.269977 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 396/777] 10.222.0.52 () {56 vars in 1112 bytes} [Sun Sep 25 00:42:37 2022] GET /identity/v3/services?type=network => generated 350 bytes in 266 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:37.278998 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a0aefa7-ec74-493e-9dae-71af98ec3f7e None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:37.315007 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-584f216c-f947-445f-a257-559c8203ae69 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:37.396019 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a9c5dc-dd90-4399-af3d-944786a22e20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:37.396665 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a9c5dc-dd90-4399-af3d-944786a22e20 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:37.396941 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a9c5dc-dd90-4399-af3d-944786a22e20 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:37.397284 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a9c5dc-dd90-4399-af3d-944786a22e20 None admin] PATH_INFO: `/v3/services` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:37.397917 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a9c5dc-dd90-4399-af3d-944786a22e20 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:37.399460 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a9c5dc-dd90-4399-af3d-944786a22e20 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:37.411658 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 382/778] 10.222.0.52 () {56 vars in 1087 bytes} [Sun Sep 25 00:42:37 2022] GET /identity/v3/services => generated 772 bytes in 341 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:37.460936 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a0aefa7-ec74-493e-9dae-71af98ec3f7e None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:37.516249 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43368045-e184-4e6c-ac3c-8bf0a37d9f7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:37.517187 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43368045-e184-4e6c-ac3c-8bf0a37d9f7c None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:37.517675 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43368045-e184-4e6c-ac3c-8bf0a37d9f7c None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:37.518122 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43368045-e184-4e6c-ac3c-8bf0a37d9f7c None admin] PATH_INFO: `/v3/endpoints` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:37.519361 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43368045-e184-4e6c-ac3c-8bf0a37d9f7c None admin] RBAC: Adding query filter params (interface=public, service_id=6076d868c0114a8a92877b12a3da865b, region_id=RegionOne) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:37.519872 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43368045-e184-4e6c-ac3c-8bf0a37d9f7c None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:37.522640 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43368045-e184-4e6c-ac3c-8bf0a37d9f7c None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:37.548751 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 397/779] 10.222.0.52 () {56 vars in 1251 bytes} [Sun Sep 25 00:42:37 2022] GET /identity/v3/endpoints?service_id=6076d868c0114a8a92877b12a3da865b&interface=public®ion_id=RegionOne => generated 194 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:37.558552 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a9c5dc-dd90-4399-af3d-944786a22e20 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:37.791851 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a9c5dc-dd90-4399-af3d-944786a22e20 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:37.845096 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:37.860344 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ba6c40-4689-4ee0-b7cb-2b96fe3561dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:37.861382 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ba6c40-4689-4ee0-b7cb-2b96fe3561dd None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:37.861847 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ba6c40-4689-4ee0-b7cb-2b96fe3561dd None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:37.862265 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ba6c40-4689-4ee0-b7cb-2b96fe3561dd None admin] PATH_INFO: `/v3/services` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:37.863578 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94ba6c40-4689-4ee0-b7cb-2b96fe3561dd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:37.866565 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94ba6c40-4689-4ee0-b7cb-2b96fe3561dd None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:37.876847 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 383/780] 10.222.0.52 () {56 vars in 1087 bytes} [Sun Sep 25 00:42:37 2022] GET /identity/v3/services => generated 772 bytes in 319 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:37.885865 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b8dd89-a34d-4c11-8f0d-9c8ae33428c0 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:37.886743 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b8dd89-a34d-4c11-8f0d-9c8ae33428c0 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:37.887157 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b8dd89-a34d-4c11-8f0d-9c8ae33428c0 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:37.890471 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 398/781] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:42:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:42:37.896496 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9265de-a808-43c2-a129-8eeefba85417 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:37.897578 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9265de-a808-43c2-a129-8eeefba85417 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:37.898058 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9265de-a808-43c2-a129-8eeefba85417 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:37.934057 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-7d9265de-a808-43c2-a129-8eeefba85417 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:42:37.935919 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-7d9265de-a808-43c2-a129-8eeefba85417 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:42:38.088887 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 384/782] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:42:37 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:42:38.319204 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f60ce7-ffb1-43c1-90c9-e613ac026cc4 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:38.319698 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f60ce7-ffb1-43c1-90c9-e613ac026cc4 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:38.320056 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f60ce7-ffb1-43c1-90c9-e613ac026cc4 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:38.322097 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 399/783] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:42:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:42:38.331054 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0556ff11-e64e-4044-8b8f-00da440f0509 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:38.331611 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0556ff11-e64e-4044-8b8f-00da440f0509 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:38.332157 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0556ff11-e64e-4044-8b8f-00da440f0509 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:38.364733 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-0556ff11-e64e-4044-8b8f-00da440f0509 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:42:38.367160 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-0556ff11-e64e-4044-8b8f-00da440f0509 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:42:38.435600 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f60ce7-ffb1-43c1-90c9-e613ac026cc4 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:38.496366 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 385/784] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:42:38 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 25 00:42:38.596285 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:38.707335 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f60ce7-ffb1-43c1-90c9-e613ac026cc4 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:38.759181 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1092cf9-d2bd-4972-bf02-fecbc7906c53 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:38.759772 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1092cf9-d2bd-4972-bf02-fecbc7906c53 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:38.760253 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1092cf9-d2bd-4972-bf02-fecbc7906c53 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:38.762187 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 386/785] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:42:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 00:42:38.775624 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf2a950-2525-4f96-a25e-f5a79a15b61e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:38.776270 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf2a950-2525-4f96-a25e-f5a79a15b61e None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:38.776540 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf2a950-2525-4f96-a25e-f5a79a15b61e None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:38.776803 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf2a950-2525-4f96-a25e-f5a79a15b61e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:38.782162 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7d1a37-5c56-4f3b-92b2-1f7d08457033 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:38.782621 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7d1a37-5c56-4f3b-92b2-1f7d08457033 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:38.783350 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7d1a37-5c56-4f3b-92b2-1f7d08457033 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:38.797290 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cf2a950-2525-4f96-a25e-f5a79a15b61e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:42:38.802242 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-5cf2a950-2525-4f96-a25e-f5a79a15b61e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:42:38.802242 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:38.802242 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:38.802242 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:38.802242 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:38.802242 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:38.802242 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:38.802242 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:38.802242 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:38.802242 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:38.802242 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:38.802242 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:38.802242 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:42:38.802242 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:42:38.802242 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:42:38.802242 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:42:38.804089 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:42:38.804089 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:42:38.804089 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:42:38.804089 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:42:38.804089 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:38.804089 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:38.804089 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:38.804089 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:38.804089 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:38.804089 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:38.804089 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:38.804089 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:38.804089 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:38.804089 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:38.804089 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:38.804089 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:38.805727 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:38.805727 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:38.805727 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:38.805727 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:38.805727 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:42:38.805727 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:42:38.805727 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:42:38.805727 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:42:38.805727 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:42:38.805727 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:42:38.805727 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:42:38.805727 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:38.813777 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 400/786] 10.222.0.52 () {56 vars in 1096 bytes} [Sun Sep 25 00:42:38 2022] GET /identity/v3/roles/admin => generated 83 bytes in 374 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:38.817565 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf2a950-2525-4f96-a25e-f5a79a15b61e None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:38.826263 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-0c7d1a37-5c56-4f3b-92b2-1f7d08457033 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:42:38.827469 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-0c7d1a37-5c56-4f3b-92b2-1f7d08457033 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:42:38.977643 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:39.023571 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 387/787] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:42:38 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 241 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:42:39.024600 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c7d1a37-5c56-4f3b-92b2-1f7d08457033 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:39.082931 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf2a950-2525-4f96-a25e-f5a79a15b61e None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:39.101743 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:39.144879 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-885c9af5-e126-4a7d-983e-f2b557521852 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:39.146667 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885c9af5-e126-4a7d-983e-f2b557521852 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:39.147538 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885c9af5-e126-4a7d-983e-f2b557521852 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:39.147817 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885c9af5-e126-4a7d-983e-f2b557521852 None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:39.148518 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-885c9af5-e126-4a7d-983e-f2b557521852 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:39.150544 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-885c9af5-e126-4a7d-983e-f2b557521852 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:39.152835 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-885c9af5-e126-4a7d-983e-f2b557521852 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:39.173232 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 401/788] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:42:38 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 357 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:42:39.180030 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-885c9af5-e126-4a7d-983e-f2b557521852 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:39.262090 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c7d1a37-5c56-4f3b-92b2-1f7d08457033 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:39.345846 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42245701-ed74-4114-8a83-420ef1bc7282 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:39.346498 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42245701-ed74-4114-8a83-420ef1bc7282 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:39.346765 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42245701-ed74-4114-8a83-420ef1bc7282 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:39.347031 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42245701-ed74-4114-8a83-420ef1bc7282 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:39.360922 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42245701-ed74-4114-8a83-420ef1bc7282 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:42:39.367005 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-42245701-ed74-4114-8a83-420ef1bc7282 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 25 00:42:39.367005 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:39.367005 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:39.367005 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:39.367005 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:39.367005 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:39.367005 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:39.367005 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:39.367005 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:39.367005 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:39.367005 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:39.367005 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:39.367005 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:42:39.367005 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:42:39.367005 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:42:39.367005 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:42:39.368019 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:42:39.368019 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:42:39.368019 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:42:39.368019 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:42:39.368019 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:39.368019 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:39.368019 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:39.368019 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:39.368019 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:39.368019 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:39.368019 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:39.368019 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:39.368019 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:39.368019 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:39.368019 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:39.368019 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:39.368887 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:39.368887 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:39.368887 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:39.368887 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:39.368887 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:42:39.368887 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:42:39.368887 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:42:39.368887 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:42:39.368887 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:42:39.368887 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:42:39.368887 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 25 00:42:39.368887 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:39.368887 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 388/789] 10.222.0.52 () {56 vars in 1102 bytes} [Sun Sep 25 00:42:39 2022] GET /identity/v3/roles/service => generated 85 bytes in 344 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:39.369611 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42245701-ed74-4114-8a83-420ef1bc7282 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:39.419800 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:39.474173 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-885c9af5-e126-4a7d-983e-f2b557521852 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:39.485354 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42245701-ed74-4114-8a83-420ef1bc7282 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:39.523466 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae00293e-08e2-41ba-aa8d-98f1d6a7a364 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:39.524457 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae00293e-08e2-41ba-aa8d-98f1d6a7a364 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:39.524942 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae00293e-08e2-41ba-aa8d-98f1d6a7a364 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:39.525412 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae00293e-08e2-41ba-aa8d-98f1d6a7a364 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:39.526834 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31e7210c-e4cf-4e16-8498-623b03c9c3b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:39.527578 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31e7210c-e4cf-4e16-8498-623b03c9c3b2 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:39.528070 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31e7210c-e4cf-4e16-8498-623b03c9c3b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:39.528559 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31e7210c-e4cf-4e16-8498-623b03c9c3b2 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:39.539537 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae00293e-08e2-41ba-aa8d-98f1d6a7a364 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:39.540991 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae00293e-08e2-41ba-aa8d-98f1d6a7a364 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:39.542217 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31e7210c-e4cf-4e16-8498-623b03c9c3b2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:42:39.547827 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-31e7210c-e4cf-4e16-8498-623b03c9c3b2 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:42:39.547827 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:39.547827 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:39.547827 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:39.547827 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:39.547827 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:39.547827 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:39.547827 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:39.547827 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:39.547827 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:39.547827 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:39.547827 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:39.547827 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:42:39.547827 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:42:39.547827 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:42:39.547827 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:42:39.548834 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:42:39.548834 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:42:39.548834 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:42:39.548834 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:42:39.548834 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:39.548834 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:39.548834 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:39.548834 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:39.548834 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:39.548834 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:39.548834 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:39.548834 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:39.548834 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:39.548834 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:39.548834 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:39.548834 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:39.549708 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:39.549708 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:39.549708 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:39.549708 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:39.549708 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:42:39.549708 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:42:39.549708 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:42:39.549708 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:42:39.549708 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:42:39.549708 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:42:39.549708 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:42:39.549708 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:39.556859 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-ae00293e-08e2-41ba-aa8d-98f1d6a7a364 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:39.556859 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:39.556859 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:39.556859 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:39.556859 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:39.556859 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:39.556859 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:39.556859 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:39.556859 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:39.556859 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:39.556859 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:39.556859 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:39.556859 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:39.556859 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:39.556859 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:39.556859 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:39.557883 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:39.557883 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:39.557883 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:39.557883 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:39.557883 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:39.557883 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:39.557883 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:39.557883 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:39.557883 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:39.557883 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:39.557883 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:39.557883 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:39.557883 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:39.557883 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:39.557883 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:39.557883 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:39.557883 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:39.558798 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:39.558798 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:39.558798 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:39.558970 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 389/790] 10.222.0.52 () {56 vars in 1096 bytes} [Sun Sep 25 00:42:39 2022] GET /identity/v3/roles/admin => generated 83 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:39.558970 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31e7210c-e4cf-4e16-8498-623b03c9c3b2 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:39.561007 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 402/791] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 381 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:42:39.561998 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae00293e-08e2-41ba-aa8d-98f1d6a7a364 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:39.596418 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:39.685726 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31e7210c-e4cf-4e16-8498-623b03c9c3b2 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:39.687601 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:39.697871 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae00293e-08e2-41ba-aa8d-98f1d6a7a364 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:39.729652 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a79437a-7b47-493f-9423-0325a496f920 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:39.730301 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a79437a-7b47-493f-9423-0325a496f920 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:39.730599 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a79437a-7b47-493f-9423-0325a496f920 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:39.731185 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a79437a-7b47-493f-9423-0325a496f920 None admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:39.731620 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a79437a-7b47-493f-9423-0325a496f920 None admin] RBAC: Adding query filter params (name=service) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:39.731925 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a79437a-7b47-493f-9423-0325a496f920 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:39.733330 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a79437a-7b47-493f-9423-0325a496f920 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:39.743309 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 390/792] 10.222.0.52 () {56 vars in 1103 bytes} [Sun Sep 25 00:42:39 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:39.744149 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a79437a-7b47-493f-9423-0325a496f920 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:39.749226 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f37a3aaf-e240-4857-8b71-edac136a48ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:39.749932 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37a3aaf-e240-4857-8b71-edac136a48ae None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:39.750214 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37a3aaf-e240-4857-8b71-edac136a48ae None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:39.750480 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37a3aaf-e240-4857-8b71-edac136a48ae None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:39.751155 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f37a3aaf-e240-4857-8b71-edac136a48ae None admin] RBAC: Adding query filter params (name=admin) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:39.751453 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f37a3aaf-e240-4857-8b71-edac136a48ae None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:39.752826 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f37a3aaf-e240-4857-8b71-edac136a48ae None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:39.769214 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 403/793] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:42:39 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:39.770153 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f37a3aaf-e240-4857-8b71-edac136a48ae None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:39.869189 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:39.878936 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a79437a-7b47-493f-9423-0325a496f920 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:39.892225 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f37a3aaf-e240-4857-8b71-edac136a48ae None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:39.926903 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f252827-4089-4b8a-ae0c-ff768c6d1da4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:39.927985 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f252827-4089-4b8a-ae0c-ff768c6d1da4 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:39.928981 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f252827-4089-4b8a-ae0c-ff768c6d1da4 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:39.929780 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f252827-4089-4b8a-ae0c-ff768c6d1da4 None admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:39.930927 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f252827-4089-4b8a-ae0c-ff768c6d1da4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:39.931482 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f252827-4089-4b8a-ae0c-ff768c6d1da4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:39.933140 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f252827-4089-4b8a-ae0c-ff768c6d1da4 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:39.934634 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75dc29dd-bcd6-49e4-ac4a-a52304e60260 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:39.935340 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75dc29dd-bcd6-49e4-ac4a-a52304e60260 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:39.935683 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75dc29dd-bcd6-49e4-ac4a-a52304e60260 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:39.936022 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75dc29dd-bcd6-49e4-ac4a-a52304e60260 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:39.947584 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75dc29dd-bcd6-49e4-ac4a-a52304e60260 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:39.949209 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75dc29dd-bcd6-49e4-ac4a-a52304e60260 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:39.949649 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 391/794] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:39.950748 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f252827-4089-4b8a-ae0c-ff768c6d1da4 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:39.968921 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-75dc29dd-bcd6-49e4-ac4a-a52304e60260 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:39.968921 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:39.968921 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:39.968921 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:39.968921 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:39.968921 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:39.968921 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:39.968921 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:39.968921 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:39.968921 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:39.968921 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:39.968921 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:39.968921 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:39.968921 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:39.968921 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:39.968921 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:39.970000 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:39.970000 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:39.970000 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:39.970000 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:39.970000 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:39.970000 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:39.970000 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:39.970000 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:39.970000 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:39.970000 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:39.970000 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:39.970000 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:39.970000 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:39.970000 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:39.970000 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:39.970000 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:39.970000 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:39.970935 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:39.970935 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:39.970935 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:39.970935 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 404/795] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:39.971784 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75dc29dd-bcd6-49e4-ac4a-a52304e60260 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:39.985020 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:40.079785 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f252827-4089-4b8a-ae0c-ff768c6d1da4 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:40.119912 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75dc29dd-bcd6-49e4-ac4a-a52304e60260 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:40.161841 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-806f2cf6-1b4d-428b-bac5-62e6987c3868 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:40.162496 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806f2cf6-1b4d-428b-bac5-62e6987c3868 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:40.163157 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806f2cf6-1b4d-428b-bac5-62e6987c3868 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:40.163704 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806f2cf6-1b4d-428b-bac5-62e6987c3868 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:40.185209 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-806f2cf6-1b4d-428b-bac5-62e6987c3868 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:40.188311 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-806f2cf6-1b4d-428b-bac5-62e6987c3868 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:40.192188 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae609f83-ab3d-41f8-a952-6134f6ebf7e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:40.193749 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae609f83-ab3d-41f8-a952-6134f6ebf7e2 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:40.195068 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae609f83-ab3d-41f8-a952-6134f6ebf7e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:40.195705 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae609f83-ab3d-41f8-a952-6134f6ebf7e2 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:40.199452 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:40.212751 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae609f83-ab3d-41f8-a952-6134f6ebf7e2 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:40.215917 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae609f83-ab3d-41f8-a952-6134f6ebf7e2 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:40.223044 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-806f2cf6-1b4d-428b-bac5-62e6987c3868 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:40.223044 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:40.223044 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:40.223044 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:40.223044 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:40.223044 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:40.223044 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:40.223044 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:40.223044 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:40.223044 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:40.223044 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:40.223044 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:40.223044 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:40.223044 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:40.223044 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:40.223044 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:40.224885 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:40.224885 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:40.224885 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:40.224885 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:40.224885 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:40.224885 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:40.224885 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:40.224885 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:40.224885 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:40.224885 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:40.224885 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:40.224885 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:40.224885 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:40.224885 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:40.224885 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:40.224885 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:40.224885 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:40.227524 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:40.227524 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:40.227524 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:40.233777 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 392/796] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 276 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:40.235359 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-806f2cf6-1b4d-428b-bac5-62e6987c3868 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:40.246006 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-ae609f83-ab3d-41f8-a952-6134f6ebf7e2 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 25 00:42:40.246006 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:40.246006 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:40.246006 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:40.246006 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:40.246006 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:40.246006 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:40.246006 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:40.246006 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:40.246006 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:40.246006 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:40.246006 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:40.246006 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 00:42:40.246006 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 00:42:40.246006 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 00:42:40.246006 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 00:42:40.247842 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:40.247842 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:40.247842 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 00:42:40.247842 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:40.247842 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 00:42:40.247842 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:40.247842 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:40.247842 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:40.247842 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:40.247842 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:40.247842 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:40.247842 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:40.247842 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:40.247842 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:40.247842 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:40.247842 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:40.247842 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:40.247842 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:40.250048 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:40.250048 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:40.250048 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 00:42:40.250048 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 00:42:40.250048 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 00:42:40.250048 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 00:42:40.250048 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 00:42:40.250048 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 00:42:40.250048 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 25 00:42:40.250048 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:40.254578 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 405/797] 10.222.0.52 () {56 vars in 1093 bytes} [Sun Sep 25 00:42:39 2022] GET /identity/v3/users/nova => generated 82 bytes in 283 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:40.256912 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae609f83-ab3d-41f8-a952-6134f6ebf7e2 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:40.354939 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:40.399643 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-806f2cf6-1b4d-428b-bac5-62e6987c3868 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:40.414213 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae609f83-ab3d-41f8-a952-6134f6ebf7e2 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:40.440679 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a8bfe5f-9ad6-4880-ae84-1a0ac0eec4d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:40.441347 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a8bfe5f-9ad6-4880-ae84-1a0ac0eec4d7 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:40.441625 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a8bfe5f-9ad6-4880-ae84-1a0ac0eec4d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:40.441892 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a8bfe5f-9ad6-4880-ae84-1a0ac0eec4d7 None admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:40.442531 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a8bfe5f-9ad6-4880-ae84-1a0ac0eec4d7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:40.442829 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a8bfe5f-9ad6-4880-ae84-1a0ac0eec4d7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:40.444820 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a8bfe5f-9ad6-4880-ae84-1a0ac0eec4d7 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:40.459124 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 393/798] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:40.459124 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a8bfe5f-9ad6-4880-ae84-1a0ac0eec4d7 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:40.467749 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48ab262a-0678-4dfc-b8a8-c24af9cd4a45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:40.468709 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ab262a-0678-4dfc-b8a8-c24af9cd4a45 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:40.469207 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ab262a-0678-4dfc-b8a8-c24af9cd4a45 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:40.469883 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ab262a-0678-4dfc-b8a8-c24af9cd4a45 None admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:40.471732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48ab262a-0678-4dfc-b8a8-c24af9cd4a45 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:40.472312 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48ab262a-0678-4dfc-b8a8-c24af9cd4a45 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:40.474941 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48ab262a-0678-4dfc-b8a8-c24af9cd4a45 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:40.494312 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 406/799] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:40.497501 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48ab262a-0678-4dfc-b8a8-c24af9cd4a45 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:40.564551 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:40.600792 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a8bfe5f-9ad6-4880-ae84-1a0ac0eec4d7 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:40.621169 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48ab262a-0678-4dfc-b8a8-c24af9cd4a45 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:40.637547 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-113cd913-7cee-4d1d-87f8-42e45ee4eac8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:40.638188 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113cd913-7cee-4d1d-87f8-42e45ee4eac8 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:40.638465 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113cd913-7cee-4d1d-87f8-42e45ee4eac8 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:40.638729 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113cd913-7cee-4d1d-87f8-42e45ee4eac8 None admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:40.639452 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-113cd913-7cee-4d1d-87f8-42e45ee4eac8 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:40.639781 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-113cd913-7cee-4d1d-87f8-42e45ee4eac8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:40.641180 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-113cd913-7cee-4d1d-87f8-42e45ee4eac8 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:40.644121 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:40.660739 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d47e0c0-0faf-4d33-ae36-ef9112199597 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:40.666001 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d47e0c0-0faf-4d33-ae36-ef9112199597 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:40.666183 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 394/800] 10.222.0.52 () {56 vars in 1133 bytes} [Sun Sep 25 00:42:40 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:40.666281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d47e0c0-0faf-4d33-ae36-ef9112199597 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:40.666281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d47e0c0-0faf-4d33-ae36-ef9112199597 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:40.666434 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554a97ba-0e2a-4d3c-828b-55ac7a2b8aa7 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:40.666434 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554a97ba-0e2a-4d3c-828b-55ac7a2b8aa7 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:40.666434 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554a97ba-0e2a-4d3c-828b-55ac7a2b8aa7 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:40.666434 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 395/801] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:42:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:42:40.666434 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554a97ba-0e2a-4d3c-828b-55ac7a2b8aa7 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:40.680495 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d47e0c0-0faf-4d33-ae36-ef9112199597 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:40.682122 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d47e0c0-0faf-4d33-ae36-ef9112199597 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:40.696685 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-2d47e0c0-0faf-4d33-ae36-ef9112199597 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 25 00:42:40.696685 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:40.696685 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:40.696685 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:40.696685 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:40.696685 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:40.696685 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:40.696685 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:40.696685 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:40.696685 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:40.696685 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:40.696685 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:40.696685 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 00:42:40.696685 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 00:42:40.696685 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 00:42:40.696685 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 00:42:40.698033 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:40.698033 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:40.698033 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 00:42:40.698033 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:40.698033 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 00:42:40.698033 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:40.698033 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:40.698033 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:40.698033 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:40.698033 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:40.698033 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:40.698033 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:40.698033 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:40.698033 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:40.698033 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:40.698033 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:40.698033 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:40.698033 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:40.699195 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:40.699195 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:40.699195 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 00:42:40.699195 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 00:42:40.699195 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 00:42:40.699195 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 00:42:40.699195 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 00:42:40.699195 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 00:42:40.699195 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 25 00:42:40.699195 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:40.702618 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 407/802] 10.222.0.52 () {56 vars in 1117 bytes} [Sun Sep 25 00:42:40 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:40.703574 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d47e0c0-0faf-4d33-ae36-ef9112199597 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:40.785546 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554a97ba-0e2a-4d3c-828b-55ac7a2b8aa7 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:40.813510 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d47e0c0-0faf-4d33-ae36-ef9112199597 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:40.822725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f36ebb-25ac-488d-aa1b-d3c1de6528d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:40.823352 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f36ebb-25ac-488d-aa1b-d3c1de6528d4 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:40.823622 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f36ebb-25ac-488d-aa1b-d3c1de6528d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:40.823888 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f36ebb-25ac-488d-aa1b-d3c1de6528d4 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:40.829217 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:40.831446 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f36ebb-25ac-488d-aa1b-d3c1de6528d4 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:40.832951 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f36ebb-25ac-488d-aa1b-d3c1de6528d4 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:40.845633 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-54f36ebb-25ac-488d-aa1b-d3c1de6528d4 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 25 00:42:40.845633 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:40.845633 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:40.845633 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:40.845633 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:40.845633 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:40.845633 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:40.845633 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:40.845633 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:40.845633 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:40.845633 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:40.845633 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:40.845633 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 00:42:40.845633 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 00:42:40.845633 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 00:42:40.845633 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 00:42:40.846642 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:40.846642 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:40.846642 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 00:42:40.846642 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:40.846642 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 00:42:40.846642 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:40.846642 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:40.846642 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:40.846642 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:40.846642 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:40.846642 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:40.846642 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:40.846642 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:40.846642 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:40.846642 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:40.846642 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:40.846642 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:40.846642 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:40.847637 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:40.847637 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:40.847637 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 00:42:40.847637 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 00:42:40.847637 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 00:42:40.847637 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 00:42:40.847637 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 00:42:40.847637 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 00:42:40.847637 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 25 00:42:40.847637 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:40.847637 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 396/803] 10.222.0.52 () {56 vars in 1096 bytes} [Sun Sep 25 00:42:40 2022] GET /identity/v3/users/swift => generated 83 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:40.848234 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-842cf6c5-e5e0-47c8-82b2-0771cd49ce7c None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:40.848234 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-842cf6c5-e5e0-47c8-82b2-0771cd49ce7c None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:40.849030 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-842cf6c5-e5e0-47c8-82b2-0771cd49ce7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:40.857561 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c581fb-02d3-4d62-a0b3-4230fe4eefa8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:40.858174 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c581fb-02d3-4d62-a0b3-4230fe4eefa8 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:40.858452 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c581fb-02d3-4d62-a0b3-4230fe4eefa8 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:40.858719 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c581fb-02d3-4d62-a0b3-4230fe4eefa8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:40.869293 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-842cf6c5-e5e0-47c8-82b2-0771cd49ce7c None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:42:40.870479 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-842cf6c5-e5e0-47c8-82b2-0771cd49ce7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:42:40.930368 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:40.930974 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25c581fb-02d3-4d62-a0b3-4230fe4eefa8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:40.932819 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25c581fb-02d3-4d62-a0b3-4230fe4eefa8 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:40.957303 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 397/804] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:42:40 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:42:40.957303 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-842cf6c5-e5e0-47c8-82b2-0771cd49ce7c None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:41.059570 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-842cf6c5-e5e0-47c8-82b2-0771cd49ce7c None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:41.096989 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 408/805] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:42:40 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 394 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 25 00:42:41.098413 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c581fb-02d3-4d62-a0b3-4230fe4eefa8 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:41.105035 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa3b283-4128-40d4-97c3-f87dce527e3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:41.105808 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa3b283-4128-40d4-97c3-f87dce527e3e None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:41.106148 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa3b283-4128-40d4-97c3-f87dce527e3e None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:41.106489 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa3b283-4128-40d4-97c3-f87dce527e3e None admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:41.107399 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa3b283-4128-40d4-97c3-f87dce527e3e None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:41.107768 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa3b283-4128-40d4-97c3-f87dce527e3e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:41.109911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa3b283-4128-40d4-97c3-f87dce527e3e None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:41.118411 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:41.137371 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 398/806] 10.222.0.52 () {56 vars in 1149 bytes} [Sun Sep 25 00:42:40 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:41.138240 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa3b283-4128-40d4-97c3-f87dce527e3e None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:41.199205 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:41.233071 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c581fb-02d3-4d62-a0b3-4230fe4eefa8 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:41.280511 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d68584f-dc8b-4ad6-8f88-bb1cc4c99bc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:41.281133 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d68584f-dc8b-4ad6-8f88-bb1cc4c99bc7 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:41.281567 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d68584f-dc8b-4ad6-8f88-bb1cc4c99bc7 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:41.281835 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d68584f-dc8b-4ad6-8f88-bb1cc4c99bc7 None admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:41.285407 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa3b283-4128-40d4-97c3-f87dce527e3e None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:41.286864 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d68584f-dc8b-4ad6-8f88-bb1cc4c99bc7 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:41.287174 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d68584f-dc8b-4ad6-8f88-bb1cc4c99bc7 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:41.288717 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d68584f-dc8b-4ad6-8f88-bb1cc4c99bc7 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:41.320977 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 409/807] 10.222.0.52 () {56 vars in 1135 bytes} [Sun Sep 25 00:42:41 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:41.320977 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d68584f-dc8b-4ad6-8f88-bb1cc4c99bc7 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:41.335367 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc7eb620-3ac4-4fa7-8636-99237ef4cbd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:41.336099 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7eb620-3ac4-4fa7-8636-99237ef4cbd3 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:41.336434 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7eb620-3ac4-4fa7-8636-99237ef4cbd3 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:41.336702 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7eb620-3ac4-4fa7-8636-99237ef4cbd3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:41.348372 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc7eb620-3ac4-4fa7-8636-99237ef4cbd3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:41.349870 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc7eb620-3ac4-4fa7-8636-99237ef4cbd3 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:41.365549 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-bc7eb620-3ac4-4fa7-8636-99237ef4cbd3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:41.365549 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:41.365549 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:41.365549 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:41.365549 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:41.365549 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:41.365549 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:41.365549 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:41.365549 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:41.365549 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:41.365549 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:41.365549 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:41.365549 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:41.365549 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:41.365549 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:41.365549 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:41.367207 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:41.367207 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:41.367207 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:41.367207 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:41.367207 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:41.367207 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:41.367207 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:41.367207 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:41.367207 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:41.367207 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:41.367207 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:41.367207 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:41.367207 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:41.367207 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:41.367207 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:41.367207 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:41.367207 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:41.368131 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:41.368131 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:41.368131 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:41.368131 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 399/808] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:41.368358 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc7eb620-3ac4-4fa7-8636-99237ef4cbd3 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:41.404179 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:41.434517 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d68584f-dc8b-4ad6-8f88-bb1cc4c99bc7 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:41.464095 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:41.471035 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc7eb620-3ac4-4fa7-8636-99237ef4cbd3 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:41.473134 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feaf6bd7-adb0-4913-8292-42de42c0cff7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:41.473807 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feaf6bd7-adb0-4913-8292-42de42c0cff7 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:41.474081 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feaf6bd7-adb0-4913-8292-42de42c0cff7 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:41.474345 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feaf6bd7-adb0-4913-8292-42de42c0cff7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:41.484811 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feaf6bd7-adb0-4913-8292-42de42c0cff7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:41.486274 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feaf6bd7-adb0-4913-8292-42de42c0cff7 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:41.502694 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-feaf6bd7-adb0-4913-8292-42de42c0cff7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:41.502694 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:41.502694 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:41.502694 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:41.502694 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:41.502694 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:41.502694 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:41.502694 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:41.502694 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:41.502694 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:41.502694 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:41.502694 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:41.502694 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:41.502694 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:41.502694 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:41.502694 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:41.503841 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:41.503841 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:41.503841 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:41.503841 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:41.503841 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:41.503841 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:41.503841 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:41.503841 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:41.503841 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:41.503841 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:41.503841 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:41.503841 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:41.503841 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:41.503841 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:41.503841 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:41.503841 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:41.503841 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:41.504746 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:41.504746 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:41.504746 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:41.504746 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 410/809] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:41.504746 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feaf6bd7-adb0-4913-8292-42de42c0cff7 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:41.514933 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083c6a3b-d0b5-4f69-a743-f44ae72dc510 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:41.515747 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083c6a3b-d0b5-4f69-a743-f44ae72dc510 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:41.516564 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083c6a3b-d0b5-4f69-a743-f44ae72dc510 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:41.516930 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083c6a3b-d0b5-4f69-a743-f44ae72dc510 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:41.517765 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-083c6a3b-d0b5-4f69-a743-f44ae72dc510 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:41.519633 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-083c6a3b-d0b5-4f69-a743-f44ae72dc510 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:41.532521 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-083c6a3b-d0b5-4f69-a743-f44ae72dc510 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 25 00:42:41.532521 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:41.532521 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 00:42:41.532521 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 00:42:41.532521 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 00:42:41.532521 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 00:42:41.532521 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 00:42:41.532521 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:42:41.532521 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 25 00:42:41.532521 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:41.532521 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 00:42:41.532521 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:41.532521 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:41.532521 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:41.532521 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:41.532521 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:41.532521 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:41.533637 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:41.533637 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:41.533637 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:41.533637 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:41.533637 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:41.533637 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:41.533637 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 00:42:41.533637 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 00:42:41.533637 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 00:42:41.533637 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 00:42:41.533637 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:41.533637 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:41.533637 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:41.533637 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:41.533637 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:41.533637 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:41.534500 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:41.534500 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:41.534500 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:41.534500 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:41.534500 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:41.534500 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:41.534500 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:41.534500 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:41.534500 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:41.534500 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:41.534500 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 00:42:41.534500 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:42:41.534500 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 25 00:42:41.534500 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:41.534500 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 400/810] 10.222.0.52 () {56 vars in 1129 bytes} [Sun Sep 25 00:42:41 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:41.535302 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083c6a3b-d0b5-4f69-a743-f44ae72dc510 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:41.627933 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feaf6bd7-adb0-4913-8292-42de42c0cff7 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:41.654481 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083c6a3b-d0b5-4f69-a743-f44ae72dc510 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:41.656823 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:41.670855 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09375309-1e8a-4399-b7eb-50832c3ddf0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:41.671600 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09375309-1e8a-4399-b7eb-50832c3ddf0b None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:41.672013 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09375309-1e8a-4399-b7eb-50832c3ddf0b None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:41.672399 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09375309-1e8a-4399-b7eb-50832c3ddf0b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:41.687079 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09375309-1e8a-4399-b7eb-50832c3ddf0b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:41.687079 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09375309-1e8a-4399-b7eb-50832c3ddf0b None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:41.693431 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f4b036d-4062-4cef-9f0f-ca529cc744ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:41.694606 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4b036d-4062-4cef-9f0f-ca529cc744ff None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:41.694875 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4b036d-4062-4cef-9f0f-ca529cc744ff None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:41.695154 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4b036d-4062-4cef-9f0f-ca529cc744ff None admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:41.695801 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f4b036d-4062-4cef-9f0f-ca529cc744ff None admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:41.696101 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f4b036d-4062-4cef-9f0f-ca529cc744ff None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:41.697499 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f4b036d-4062-4cef-9f0f-ca529cc744ff None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:41.702808 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-09375309-1e8a-4399-b7eb-50832c3ddf0b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:41.702808 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:41.702808 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:41.702808 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:41.702808 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:41.702808 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:41.702808 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:41.702808 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:41.702808 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:41.702808 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:41.702808 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:41.702808 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:41.702808 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:41.702808 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:41.702808 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:41.702808 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:41.703891 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:41.703891 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:41.703891 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:41.703891 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:41.703891 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:41.703891 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:41.703891 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:41.703891 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:41.703891 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:41.703891 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:41.703891 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:41.703891 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:41.703891 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:41.703891 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:41.703891 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:41.703891 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:41.703891 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:41.705203 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:41.705203 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:41.705203 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:41.705203 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 411/811] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:41.705524 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74430428-43af-4cef-8e1d-00237cf15911 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:41.705669 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74430428-43af-4cef-8e1d-00237cf15911 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:41.705969 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74430428-43af-4cef-8e1d-00237cf15911 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:41.709299 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 412/812] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:42:41 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:42:41.709299 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74430428-43af-4cef-8e1d-00237cf15911 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:41.714286 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 401/813] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:41.714957 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f4b036d-4062-4cef-9f0f-ca529cc744ff None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:41.734958 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:41.818657 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74430428-43af-4cef-8e1d-00237cf15911 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:41.824968 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f4b036d-4062-4cef-9f0f-ca529cc744ff None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:41.860949 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff08c63b-408e-47b1-95f7-0256cbc6b0ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:41.861580 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff08c63b-408e-47b1-95f7-0256cbc6b0ac None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:41.861860 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff08c63b-408e-47b1-95f7-0256cbc6b0ac None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:41.861968 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ece34eb-44a3-4465-beca-bb02fa901d60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:41.862122 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff08c63b-408e-47b1-95f7-0256cbc6b0ac None admin] PATH_INFO: `/v3/services` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:41.862519 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ece34eb-44a3-4465-beca-bb02fa901d60 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:41.862744 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff08c63b-408e-47b1-95f7-0256cbc6b0ac None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:41.862836 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ece34eb-44a3-4465-beca-bb02fa901d60 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:41.863301 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ece34eb-44a3-4465-beca-bb02fa901d60 None admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:41.863398 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff08c63b-408e-47b1-95f7-0256cbc6b0ac None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:41.863951 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ece34eb-44a3-4465-beca-bb02fa901d60 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:41.864230 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ece34eb-44a3-4465-beca-bb02fa901d60 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:41.864665 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff08c63b-408e-47b1-95f7-0256cbc6b0ac None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:41.865728 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ece34eb-44a3-4465-beca-bb02fa901d60 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:41.869959 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 402/814] 10.222.0.52 () {56 vars in 1124 bytes} [Sun Sep 25 00:42:41 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:41.870750 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff08c63b-408e-47b1-95f7-0256cbc6b0ac None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:41.888348 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 413/815] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:41.888348 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d0e366-0f6b-4d44-be65-a3905f95cea3 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:41.888348 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d0e366-0f6b-4d44-be65-a3905f95cea3 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:41.888348 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d0e366-0f6b-4d44-be65-a3905f95cea3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:41.905999 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-25d0e366-0f6b-4d44-be65-a3905f95cea3 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:42:41.907249 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-25d0e366-0f6b-4d44-be65-a3905f95cea3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:42:41.940889 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:41.969193 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:41.989597 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff08c63b-408e-47b1-95f7-0256cbc6b0ac None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:41.989857 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 414/816] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:42:41 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:42:41.990527 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d0e366-0f6b-4d44-be65-a3905f95cea3 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:42.029879 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd070313-7548-4cb8-8bee-c7aa93290850 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:42.030508 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd070313-7548-4cb8-8bee-c7aa93290850 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:42.030789 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd070313-7548-4cb8-8bee-c7aa93290850 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:42.031056 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd070313-7548-4cb8-8bee-c7aa93290850 None admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:42.031697 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd070313-7548-4cb8-8bee-c7aa93290850 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:42.032003 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd070313-7548-4cb8-8bee-c7aa93290850 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:42.033390 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd070313-7548-4cb8-8bee-c7aa93290850 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:42.053787 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 403/817] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:42.053787 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd070313-7548-4cb8-8bee-c7aa93290850 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:42.118949 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d0e366-0f6b-4d44-be65-a3905f95cea3 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:42.171848 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58caaf28-e6ec-4145-8e6f-46fa14828975 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:42.173290 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58caaf28-e6ec-4145-8e6f-46fa14828975 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:42.173998 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58caaf28-e6ec-4145-8e6f-46fa14828975 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:42.174690 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58caaf28-e6ec-4145-8e6f-46fa14828975 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:42.188524 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58caaf28-e6ec-4145-8e6f-46fa14828975 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:42.191890 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58caaf28-e6ec-4145-8e6f-46fa14828975 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:42.213724 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-58caaf28-e6ec-4145-8e6f-46fa14828975 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:42:42.213724 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:42.213724 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:42.213724 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:42.213724 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:42.213724 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:42.213724 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:42.213724 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:42.213724 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:42.213724 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:42.213724 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:42.213724 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:42.213724 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 00:42:42.213724 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 00:42:42.213724 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 00:42:42.215396 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 00:42:42.215396 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:42.215396 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:42.215396 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:42.215396 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:42.215396 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:42.215396 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:42.215396 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:42.215396 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:42.215396 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:42.215396 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:42.215396 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:42.215396 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:42.215396 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:42.215396 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:42.215396 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:42.215396 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:42.217077 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 00:42:42.217077 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 00:42:42.217077 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 00:42:42.217077 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 00:42:42.217077 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 00:42:42.217077 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 00:42:42.217077 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:42:42.217077 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:42.218358 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd070313-7548-4cb8-8bee-c7aa93290850 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:42.221000 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 415/818] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:42:41 2022] GET /identity/v3/projects/service => generated 88 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:42.222747 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58caaf28-e6ec-4145-8e6f-46fa14828975 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:42.262441 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ed2c874-c59e-4284-8930-4c31f65b99f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:42.262950 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:42.263414 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed2c874-c59e-4284-8930-4c31f65b99f2 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:42.263414 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed2c874-c59e-4284-8930-4c31f65b99f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:42.263684 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed2c874-c59e-4284-8930-4c31f65b99f2 None admin] PATH_INFO: `/v3/services` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:42.264464 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ed2c874-c59e-4284-8930-4c31f65b99f2 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:42.264756 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ed2c874-c59e-4284-8930-4c31f65b99f2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:42.266231 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ed2c874-c59e-4284-8930-4c31f65b99f2 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:42.273190 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 404/819] 10.222.0.52 () {56 vars in 1124 bytes} [Sun Sep 25 00:42:42 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:42.273915 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ed2c874-c59e-4284-8930-4c31f65b99f2 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:42.292458 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:42.378141 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58caaf28-e6ec-4145-8e6f-46fa14828975 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:42.425435 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a37bba0-ffe9-49a1-a54e-6a0b2d35bf98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:42.426070 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a37bba0-ffe9-49a1-a54e-6a0b2d35bf98 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:42.426343 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a37bba0-ffe9-49a1-a54e-6a0b2d35bf98 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:42.426600 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ed2c874-c59e-4284-8930-4c31f65b99f2 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:42.426758 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a37bba0-ffe9-49a1-a54e-6a0b2d35bf98 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:42.434632 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a37bba0-ffe9-49a1-a54e-6a0b2d35bf98 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:42.436153 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a37bba0-ffe9-49a1-a54e-6a0b2d35bf98 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:42.449083 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-6a37bba0-ffe9-49a1-a54e-6a0b2d35bf98 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:42:42.449083 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:42.449083 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:42.449083 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:42.449083 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:42.449083 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:42.449083 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:42.449083 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:42.449083 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:42.449083 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:42.449083 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:42.449083 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:42.449083 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 00:42:42.449083 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 00:42:42.449083 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 00:42:42.450801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 00:42:42.450801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:42.450801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:42.450801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:42.450801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:42.450801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:42.450801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:42.450801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:42.450801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:42.450801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:42.450801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:42.450801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:42.450801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:42.450801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:42.450801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:42.450801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:42.450801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:42.452354 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 00:42:42.452354 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 00:42:42.452354 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 00:42:42.452354 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 00:42:42.452354 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 00:42:42.452354 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 00:42:42.452354 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:42:42.452354 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:42.452354 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 416/820] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:42:42 2022] GET /identity/v3/projects/service => generated 88 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:42.452354 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a37bba0-ffe9-49a1-a54e-6a0b2d35bf98 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:42.471187 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d1e171-ab17-4c50-8241-f51289d800c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:42.472059 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d1e171-ab17-4c50-8241-f51289d800c5 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:42.472464 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d1e171-ab17-4c50-8241-f51289d800c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:42.472912 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d1e171-ab17-4c50-8241-f51289d800c5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:42.483928 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03d1e171-ab17-4c50-8241-f51289d800c5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:42.486137 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03d1e171-ab17-4c50-8241-f51289d800c5 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:42.504006 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-03d1e171-ab17-4c50-8241-f51289d800c5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:42:42.504006 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:42.504006 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:42.504006 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:42.504006 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:42.504006 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:42.504006 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:42.504006 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:42.504006 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:42.504006 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:42.504006 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:42.504006 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:42.504006 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 00:42:42.504006 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 00:42:42.504006 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 00:42:42.506059 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 00:42:42.506059 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:42.506059 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:42.506059 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:42.506059 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:42.506059 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:42.506059 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:42.506059 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:42.506059 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:42.506059 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:42.506059 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:42.506059 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:42.506059 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:42.506059 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:42.506059 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:42.506059 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:42.506059 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:42.508845 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 00:42:42.508845 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 00:42:42.508845 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 00:42:42.508845 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 00:42:42.508845 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 00:42:42.508845 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 00:42:42.508845 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:42:42.508845 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:42.508845 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 405/821] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:42:42 2022] GET /identity/v3/projects/service => generated 88 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:42.508845 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d1e171-ab17-4c50-8241-f51289d800c5 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:42.565237 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:42.591088 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a37bba0-ffe9-49a1-a54e-6a0b2d35bf98 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:42.592618 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:42.639087 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d7eecc-c6fc-4740-87e6-2d7789063a65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:42.639706 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d7eecc-c6fc-4740-87e6-2d7789063a65 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:42.639981 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d7eecc-c6fc-4740-87e6-2d7789063a65 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:42.640240 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d7eecc-c6fc-4740-87e6-2d7789063a65 None admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:42.640931 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2d7eecc-c6fc-4740-87e6-2d7789063a65 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:42.641229 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2d7eecc-c6fc-4740-87e6-2d7789063a65 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:42.641878 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d1e171-ab17-4c50-8241-f51289d800c5 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:42.642707 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2d7eecc-c6fc-4740-87e6-2d7789063a65 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:42.656192 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 417/822] 10.222.0.52 () {56 vars in 1148 bytes} [Sun Sep 25 00:42:42 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:42.657042 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d7eecc-c6fc-4740-87e6-2d7789063a65 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:42.683733 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86765f0d-e9e6-407c-8891-91c2703db220 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:42.684522 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86765f0d-e9e6-407c-8891-91c2703db220 None admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:42.684878 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86765f0d-e9e6-407c-8891-91c2703db220 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:42.685285 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86765f0d-e9e6-407c-8891-91c2703db220 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:42.686176 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86765f0d-e9e6-407c-8891-91c2703db220 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:42.687716 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86765f0d-e9e6-407c-8891-91c2703db220 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:42.720334 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 406/823] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:42:42 2022] POST /identity/v3/endpoints => generated 346 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:42:42.720334 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86765f0d-e9e6-407c-8891-91c2703db220 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:42.783655 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d7eecc-c6fc-4740-87e6-2d7789063a65 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:42.859051 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1781593b-f827-43d1-8e22-3826499f74b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:42.859918 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1781593b-f827-43d1-8e22-3826499f74b8 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:42.860548 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1781593b-f827-43d1-8e22-3826499f74b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:42.860902 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1781593b-f827-43d1-8e22-3826499f74b8 None admin] PATH_INFO: `/v3/services/network` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:42.862643 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1781593b-f827-43d1-8e22-3826499f74b8 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:42.864695 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1781593b-f827-43d1-8e22-3826499f74b8 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:42.869945 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86765f0d-e9e6-407c-8891-91c2703db220 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:42.878000 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-1781593b-f827-43d1-8e22-3826499f74b8 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 25 00:42:42.878000 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:42.878000 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 00:42:42.878000 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 00:42:42.878000 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 00:42:42.878000 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 00:42:42.878000 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 00:42:42.878000 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:42:42.878000 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 25 00:42:42.878000 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:42.878000 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 00:42:42.878000 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:42.878000 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:42.878000 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:42.878000 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:42.878000 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:42.878000 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:42.879983 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:42.879983 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:42.879983 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:42.879983 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:42.879983 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:42.879983 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:42.879983 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 00:42:42.879983 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 00:42:42.879983 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 00:42:42.879983 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 00:42:42.879983 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:42.879983 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:42.879983 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:42.879983 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:42.879983 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:42.879983 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:42.881790 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:42.881790 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:42.881790 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:42.881790 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:42.881790 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:42.881790 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:42.881790 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:42.881790 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:42.881790 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:42.881790 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:42.881790 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 00:42:42.881790 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:42:42.881790 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 25 00:42:42.881790 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:42.881790 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 418/824] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:42:42 2022] GET /identity/v3/services/network => generated 88 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:42.883029 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1781593b-f827-43d1-8e22-3826499f74b8 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:42.897812 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:42.907953 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:42.937156 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05eae4d-2109-4d60-857a-f056cdd71556 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:42.938493 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05eae4d-2109-4d60-857a-f056cdd71556 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:42.938859 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05eae4d-2109-4d60-857a-f056cdd71556 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:42.939213 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05eae4d-2109-4d60-857a-f056cdd71556 None admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:42.940021 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b05eae4d-2109-4d60-857a-f056cdd71556 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:42.940524 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b05eae4d-2109-4d60-857a-f056cdd71556 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:42.942654 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b05eae4d-2109-4d60-857a-f056cdd71556 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:42.961829 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 407/825] 10.222.0.52 () {56 vars in 1148 bytes} [Sun Sep 25 00:42:42 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:42.962451 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05eae4d-2109-4d60-857a-f056cdd71556 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:43.043241 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1781593b-f827-43d1-8e22-3826499f74b8 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:43.083632 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbcb0969-234c-49a5-956f-ff1dd9166c3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:43.085075 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbcb0969-234c-49a5-956f-ff1dd9166c3b None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:43.085375 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbcb0969-234c-49a5-956f-ff1dd9166c3b None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:43.085642 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbcb0969-234c-49a5-956f-ff1dd9166c3b None admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:43.086462 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbcb0969-234c-49a5-956f-ff1dd9166c3b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:43.086791 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbcb0969-234c-49a5-956f-ff1dd9166c3b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:43.088305 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbcb0969-234c-49a5-956f-ff1dd9166c3b None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:43.093440 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05eae4d-2109-4d60-857a-f056cdd71556 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:43.108024 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 419/826] 10.222.0.52 () {56 vars in 1148 bytes} [Sun Sep 25 00:42:42 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:43.108024 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbcb0969-234c-49a5-956f-ff1dd9166c3b None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:43.145865 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1868cfe6-f02c-402f-ab83-0626fc72a29b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:43.146835 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1868cfe6-f02c-402f-ab83-0626fc72a29b None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:43.147798 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1868cfe6-f02c-402f-ab83-0626fc72a29b None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:43.148258 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1868cfe6-f02c-402f-ab83-0626fc72a29b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:43.150041 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1868cfe6-f02c-402f-ab83-0626fc72a29b None admin] RBAC: Adding query filter params (role.id=2c242abf6ed8410ab0a4a354ca1794fb, scope.project.id=9db0fd07fc494203b2ada13f3cd046d3, user.id=f142988a4dd14a5f96eacea45c046f85) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:43.150041 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1868cfe6-f02c-402f-ab83-0626fc72a29b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:43.152044 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1868cfe6-f02c-402f-ab83-0626fc72a29b None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:43.161846 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 408/827] 10.222.0.52 () {56 vars in 1374 bytes} [Sun Sep 25 00:42:42 2022] GET /identity/v3/role_assignments?user.id=f142988a4dd14a5f96eacea45c046f85&scope.project.id=9db0fd07fc494203b2ada13f3cd046d3&role.id=2c242abf6ed8410ab0a4a354ca1794fb => generated 608 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:43.163126 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1868cfe6-f02c-402f-ab83-0626fc72a29b None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:43.236371 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:43.275252 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:43.305110 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbcb0969-234c-49a5-956f-ff1dd9166c3b None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:43.354287 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4376e2f-d6ff-4c1c-9afe-fc906bb464a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:43.354903 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4376e2f-d6ff-4c1c-9afe-fc906bb464a4 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:43.355233 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4376e2f-d6ff-4c1c-9afe-fc906bb464a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:43.355507 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4376e2f-d6ff-4c1c-9afe-fc906bb464a4 None admin] PATH_INFO: `/v3/services` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:43.356631 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4376e2f-d6ff-4c1c-9afe-fc906bb464a4 None admin] RBAC: Adding query filter params (name=network) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:43.356631 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4376e2f-d6ff-4c1c-9afe-fc906bb464a4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:43.357964 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4376e2f-d6ff-4c1c-9afe-fc906bb464a4 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:43.367721 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 420/828] 10.222.0.52 () {56 vars in 1112 bytes} [Sun Sep 25 00:42:43 2022] GET /identity/v3/services?name=network => generated 124 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:43.367971 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4376e2f-d6ff-4c1c-9afe-fc906bb464a4 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:43.400636 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1868cfe6-f02c-402f-ab83-0626fc72a29b None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:43.443044 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e6ca427-2c73-424e-897d-361de4f86138 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:43.446274 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6ca427-2c73-424e-897d-361de4f86138 None admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:43.446274 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6ca427-2c73-424e-897d-361de4f86138 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:43.446274 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6ca427-2c73-424e-897d-361de4f86138 None admin] PATH_INFO: `/v3/projects/9db0fd07fc494203b2ada13f3cd046d3/users/3f20ffd8df994c1db1b1e7248db5b5a3/roles/a8e34e460e8a4ed7bcaa54da3c2d5a0c` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:43.467889 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e6ca427-2c73-424e-897d-361de4f86138 None admin] RBAC: Authorizing `identity:create_grant(project_id=9db0fd07fc494203b2ada13f3cd046d3, user_id=3f20ffd8df994c1db1b1e7248db5b5a3, role_id=a8e34e460e8a4ed7bcaa54da3c2d5a0c)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:43.469369 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e6ca427-2c73-424e-897d-361de4f86138 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:43.504027 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 409/829] 10.222.0.52 () {58 vars in 1439 bytes} [Sun Sep 25 00:42:43 2022] PUT /identity/v3/projects/9db0fd07fc494203b2ada13f3cd046d3/users/3f20ffd8df994c1db1b1e7248db5b5a3/roles/a8e34e460e8a4ed7bcaa54da3c2d5a0c => generated 0 bytes in 341 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:42:43.505454 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e6ca427-2c73-424e-897d-361de4f86138 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:43.520391 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4376e2f-d6ff-4c1c-9afe-fc906bb464a4 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:43.582899 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:43.595226 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-946215b1-a1cf-43ad-b712-814f3e88c107 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:43.595949 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946215b1-a1cf-43ad-b712-814f3e88c107 None admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:43.596229 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946215b1-a1cf-43ad-b712-814f3e88c107 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:43.596498 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946215b1-a1cf-43ad-b712-814f3e88c107 None admin] PATH_INFO: `/v3/projects/9db0fd07fc494203b2ada13f3cd046d3/users/da606b3906dd4171bd76fd281e03e258/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:43.601667 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-946215b1-a1cf-43ad-b712-814f3e88c107 None admin] RBAC: Authorizing `identity:create_grant(project_id=9db0fd07fc494203b2ada13f3cd046d3, user_id=da606b3906dd4171bd76fd281e03e258, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:43.603176 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-946215b1-a1cf-43ad-b712-814f3e88c107 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:43.630432 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 421/830] 10.222.0.52 () {58 vars in 1439 bytes} [Sun Sep 25 00:42:43 2022] PUT /identity/v3/projects/9db0fd07fc494203b2ada13f3cd046d3/users/da606b3906dd4171bd76fd281e03e258/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 263 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:42:43.693306 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:43.705648 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e6ca427-2c73-424e-897d-361de4f86138 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:43.766965 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8df1e127-22fe-4365-aaf4-befe6df64583 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:43.767931 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df1e127-22fe-4365-aaf4-befe6df64583 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:43.768216 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df1e127-22fe-4365-aaf4-befe6df64583 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:43.768489 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df1e127-22fe-4365-aaf4-befe6df64583 None admin] PATH_INFO: `/v3/services` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:43.769157 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8df1e127-22fe-4365-aaf4-befe6df64583 None admin] RBAC: Adding query filter params (type=network) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:43.769491 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8df1e127-22fe-4365-aaf4-befe6df64583 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:43.770939 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8df1e127-22fe-4365-aaf4-befe6df64583 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:43.783159 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 410/831] 10.222.0.52 () {56 vars in 1112 bytes} [Sun Sep 25 00:42:43 2022] GET /identity/v3/services?type=network => generated 350 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:43.789487 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-946215b1-a1cf-43ad-b712-814f3e88c107 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:43.965835 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-946215b1-a1cf-43ad-b712-814f3e88c107 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:44.015823 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf5f364e-b9a6-412e-adc0-48efceb5331b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:44.016453 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5f364e-b9a6-412e-adc0-48efceb5331b None admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:44.016760 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5f364e-b9a6-412e-adc0-48efceb5331b None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:44.021290 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5f364e-b9a6-412e-adc0-48efceb5331b None admin] PATH_INFO: `/v3/endpoints` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:44.021290 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf5f364e-b9a6-412e-adc0-48efceb5331b None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:44.021290 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf5f364e-b9a6-412e-adc0-48efceb5331b None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:44.056448 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 422/832] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:42:43 2022] POST /identity/v3/endpoints => generated 337 bytes in 268 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:42:45.426626 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fcde489-5380-4a30-a921-14444d563b29 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:45.427231 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fcde489-5380-4a30-a921-14444d563b29 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:45.427231 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fcde489-5380-4a30-a921-14444d563b29 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:45.428782 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 411/833] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:42:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:42:45.434541 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e519040-c194-4cf4-a847-84740bd98717 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:45.435107 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e519040-c194-4cf4-a847-84740bd98717 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:45.435107 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e519040-c194-4cf4-a847-84740bd98717 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:45.454496 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-3e519040-c194-4cf4-a847-84740bd98717 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:42:45.455713 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-3e519040-c194-4cf4-a847-84740bd98717 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:42:45.515418 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:45.564427 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 423/834] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:42:45 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:42:45.732833 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e75ec66-a6e6-46ce-a611-dcfaaf7c9991 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:45.732833 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e75ec66-a6e6-46ce-a611-dcfaaf7c9991 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:45.732833 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e75ec66-a6e6-46ce-a611-dcfaaf7c9991 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:45.733630 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 412/835] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:42:45 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 00:42:45.739674 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf751c37-aba1-40a5-b037-16d62b9e3dd6 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:45.740252 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf751c37-aba1-40a5-b037-16d62b9e3dd6 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:45.740823 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf751c37-aba1-40a5-b037-16d62b9e3dd6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:45.773385 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-bf751c37-aba1-40a5-b037-16d62b9e3dd6 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:42:45.775272 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-bf751c37-aba1-40a5-b037-16d62b9e3dd6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:42:45.856403 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e75ec66-a6e6-46ce-a611-dcfaaf7c9991 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:45.895149 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 424/836] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:42:45 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:42:45.969579 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e75ec66-a6e6-46ce-a611-dcfaaf7c9991 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:46.006163 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a90d1de-877f-430b-8f3c-2ec96722a934 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:46.006777 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a90d1de-877f-430b-8f3c-2ec96722a934 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:46.007055 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a90d1de-877f-430b-8f3c-2ec96722a934 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:46.007325 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a90d1de-877f-430b-8f3c-2ec96722a934 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:46.007911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a90d1de-877f-430b-8f3c-2ec96722a934 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:46.009291 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a90d1de-877f-430b-8f3c-2ec96722a934 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:46.020000 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-8a90d1de-877f-430b-8f3c-2ec96722a934 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 25 00:42:46.020000 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:46.020000 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 00:42:46.020000 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 00:42:46.020000 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 00:42:46.020000 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 00:42:46.020000 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 00:42:46.020000 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:42:46.020000 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 25 00:42:46.020000 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:46.020000 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 00:42:46.020000 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:46.020000 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:46.020000 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:46.020000 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:46.020000 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:46.020000 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:46.021142 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:46.021142 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:46.021142 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:46.021142 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:46.021142 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:46.021142 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:46.021142 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 00:42:46.021142 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 00:42:46.021142 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 00:42:46.021142 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 00:42:46.021142 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:46.021142 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:46.021142 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:46.021142 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:46.021142 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:46.021142 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:46.022680 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:46.022680 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:46.022680 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:46.022680 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:46.022680 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:46.022680 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:46.022680 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:46.022680 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:46.022680 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:46.022680 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:46.022680 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 00:42:46.022680 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:42:46.022680 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 25 00:42:46.022680 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:46.022680 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 413/837] 10.222.0.52 () {56 vars in 1114 bytes} [Sun Sep 25 00:42:45 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:46.028402 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf751c37-aba1-40a5-b037-16d62b9e3dd6 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:46.142171 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf751c37-aba1-40a5-b037-16d62b9e3dd6 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:46.144177 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:46.179830 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1e06ce-a198-4079-9cf0-ef0bc6df8376 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:46.180455 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1e06ce-a198-4079-9cf0-ef0bc6df8376 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:46.180734 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1e06ce-a198-4079-9cf0-ef0bc6df8376 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:46.181004 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1e06ce-a198-4079-9cf0-ef0bc6df8376 None admin] PATH_INFO: `/v3/services` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:46.181671 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be1e06ce-a198-4079-9cf0-ef0bc6df8376 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:46.181972 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be1e06ce-a198-4079-9cf0-ef0bc6df8376 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:46.183347 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be1e06ce-a198-4079-9cf0-ef0bc6df8376 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:46.188731 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 425/838] 10.222.0.52 () {56 vars in 1114 bytes} [Sun Sep 25 00:42:46 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:46.195593 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a90d1de-877f-430b-8f3c-2ec96722a934 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:46.201150 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1e06ce-a198-4079-9cf0-ef0bc6df8376 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:46.227391 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:46.313951 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1e06ce-a198-4079-9cf0-ef0bc6df8376 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:46.320160 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a90d1de-877f-430b-8f3c-2ec96722a934 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:46.350972 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c315fff3-9d84-4bdf-9f4f-612ab39a86b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:46.351617 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c315fff3-9d84-4bdf-9f4f-612ab39a86b4 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:46.351886 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c315fff3-9d84-4bdf-9f4f-612ab39a86b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:46.352424 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c315fff3-9d84-4bdf-9f4f-612ab39a86b4 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:46.352727 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c315fff3-9d84-4bdf-9f4f-612ab39a86b4 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:46.354113 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c315fff3-9d84-4bdf-9f4f-612ab39a86b4 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:46.363691 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fc235ca-4087-4aa9-8dd5-fe59c1cbc92a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:46.364334 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc235ca-4087-4aa9-8dd5-fe59c1cbc92a None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:46.364444 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-c315fff3-9d84-4bdf-9f4f-612ab39a86b4 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 25 00:42:46.364444 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:46.364444 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 00:42:46.364444 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 00:42:46.364444 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 00:42:46.364444 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 00:42:46.364444 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 00:42:46.364444 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:42:46.364444 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 25 00:42:46.364444 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:46.364444 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 00:42:46.364444 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:46.364444 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:46.364444 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:46.364444 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:46.364444 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:46.365751 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:46.365751 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:46.365751 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:46.365751 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:46.365751 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:46.365751 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:46.365751 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:46.365751 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 00:42:46.365751 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 00:42:46.365751 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 00:42:46.365751 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 00:42:46.365751 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:46.365751 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:46.365751 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:46.365751 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:46.365751 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:46.365751 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:46.366823 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:46.366823 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:46.366823 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:46.366823 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:46.366823 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:46.366823 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:46.366823 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:46.366823 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:46.366823 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:46.366823 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:46.366823 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 00:42:46.366823 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:42:46.366823 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 25 00:42:46.366823 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:46.367573 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc235ca-4087-4aa9-8dd5-fe59c1cbc92a None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:46.367573 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc235ca-4087-4aa9-8dd5-fe59c1cbc92a None admin] PATH_INFO: `/v3/services` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:46.367573 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fc235ca-4087-4aa9-8dd5-fe59c1cbc92a None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:46.367573 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fc235ca-4087-4aa9-8dd5-fe59c1cbc92a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:46.367573 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fc235ca-4087-4aa9-8dd5-fe59c1cbc92a None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:46.369404 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 426/839] 10.222.0.52 () {56 vars in 1132 bytes} [Sun Sep 25 00:42:46 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:42:46.370696 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb77949f-3ff6-4915-b454-40decb31a19c None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:46.370969 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb77949f-3ff6-4915-b454-40decb31a19c None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:46.371230 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb77949f-3ff6-4915-b454-40decb31a19c None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:46.373368 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 427/840] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:42:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:42:46.374063 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb77949f-3ff6-4915-b454-40decb31a19c None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:46.377520 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 414/841] 10.222.0.52 () {56 vars in 1114 bytes} [Sun Sep 25 00:42:46 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:42:46.386800 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c170533-0b83-4038-8fe6-e48e63df0b44 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:46.387113 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c170533-0b83-4038-8fe6-e48e63df0b44 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:46.387351 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c170533-0b83-4038-8fe6-e48e63df0b44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:46.421468 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-6c170533-0b83-4038-8fe6-e48e63df0b44 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:42:46.423436 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-6c170533-0b83-4038-8fe6-e48e63df0b44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:42:46.427431 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:46.524633 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 415/842] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:42:46 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:42:46.552108 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6742bb84-c269-4c41-afd8-3809535ef704 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:46.552473 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6742bb84-c269-4c41-afd8-3809535ef704 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:46.552682 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6742bb84-c269-4c41-afd8-3809535ef704 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:46.559600 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 416/843] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:42:46 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 00:42:46.561406 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc2c7a64-b897-4e15-9251-db0837369cbb None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:46.561737 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc2c7a64-b897-4e15-9251-db0837369cbb None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:46.562008 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc2c7a64-b897-4e15-9251-db0837369cbb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:46.570075 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb77949f-3ff6-4915-b454-40decb31a19c None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:46.582154 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-cc2c7a64-b897-4e15-9251-db0837369cbb None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:42:46.583341 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-cc2c7a64-b897-4e15-9251-db0837369cbb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:42:46.597279 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:46.611884 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56737880-2a13-4cd1-a7cf-1670c0ac3f5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:46.612488 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56737880-2a13-4cd1-a7cf-1670c0ac3f5c None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:46.612760 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56737880-2a13-4cd1-a7cf-1670c0ac3f5c None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:46.614057 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56737880-2a13-4cd1-a7cf-1670c0ac3f5c None admin] PATH_INFO: `/v3/services` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:46.614057 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56737880-2a13-4cd1-a7cf-1670c0ac3f5c None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:46.614394 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56737880-2a13-4cd1-a7cf-1670c0ac3f5c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:46.616878 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56737880-2a13-4cd1-a7cf-1670c0ac3f5c None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:46.622804 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 428/844] 10.222.0.52 () {56 vars in 1126 bytes} [Sun Sep 25 00:42:46 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:46.628164 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56737880-2a13-4cd1-a7cf-1670c0ac3f5c None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:46.677400 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 417/845] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:42:46 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 25 00:42:46.730031 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:46.736998 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56737880-2a13-4cd1-a7cf-1670c0ac3f5c None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:46.773378 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0f6ac7f-4e30-435d-8da5-b031e9774bb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:46.773983 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f6ac7f-4e30-435d-8da5-b031e9774bb9 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:46.774254 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f6ac7f-4e30-435d-8da5-b031e9774bb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:46.774515 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f6ac7f-4e30-435d-8da5-b031e9774bb9 None admin] PATH_INFO: `/v3/services` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:46.775131 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0f6ac7f-4e30-435d-8da5-b031e9774bb9 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:46.775425 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0f6ac7f-4e30-435d-8da5-b031e9774bb9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:46.776766 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0f6ac7f-4e30-435d-8da5-b031e9774bb9 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:46.781723 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 429/846] 10.222.0.52 () {56 vars in 1126 bytes} [Sun Sep 25 00:42:46 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:42:46.805412 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc2c7a64-b897-4e15-9251-db0837369cbb None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:46.932235 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc2c7a64-b897-4e15-9251-db0837369cbb None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:46.965720 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0f6ac7f-4e30-435d-8da5-b031e9774bb9 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:46.970732 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b75650-d87d-48f1-840a-be0710fe0964 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:46.971610 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b75650-d87d-48f1-840a-be0710fe0964 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:46.972068 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b75650-d87d-48f1-840a-be0710fe0964 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:46.972533 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b75650-d87d-48f1-840a-be0710fe0964 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:46.984721 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48b75650-d87d-48f1-840a-be0710fe0964 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:42:46.990486 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-48b75650-d87d-48f1-840a-be0710fe0964 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 25 00:42:46.990486 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:46.990486 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:46.990486 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:46.990486 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:46.990486 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:46.990486 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:46.990486 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:46.990486 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:46.990486 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:46.990486 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:46.990486 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:46.990486 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:42:46.990486 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:42:46.990486 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:42:46.990486 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:42:46.991688 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:42:46.991688 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:42:46.991688 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:42:46.991688 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:42:46.991688 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:46.991688 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:46.991688 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:46.991688 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:46.991688 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:46.991688 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:46.991688 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:46.991688 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:46.991688 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:46.991688 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:46.991688 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:46.991688 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:46.992529 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:46.992529 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:46.992529 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:46.992529 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:46.992529 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:42:46.992529 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:42:46.992529 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:42:46.992529 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:42:46.992529 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:42:46.992529 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:42:46.992529 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 25 00:42:46.992529 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:46.992529 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 418/847] 10.222.0.52 () {56 vars in 1102 bytes} [Sun Sep 25 00:42:46 2022] GET /identity/v3/roles/service => generated 85 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:46.999354 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b75650-d87d-48f1-840a-be0710fe0964 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:47.005418 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:47.097435 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0f6ac7f-4e30-435d-8da5-b031e9774bb9 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:47.139104 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b75650-d87d-48f1-840a-be0710fe0964 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:47.139515 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6957f55e-db3c-4934-a20f-0ed5ed3fc764 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:47.140154 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6957f55e-db3c-4934-a20f-0ed5ed3fc764 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:47.140428 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6957f55e-db3c-4934-a20f-0ed5ed3fc764 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:47.140809 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6957f55e-db3c-4934-a20f-0ed5ed3fc764 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:47.153376 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6957f55e-db3c-4934-a20f-0ed5ed3fc764 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:42:47.159112 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-6957f55e-db3c-4934-a20f-0ed5ed3fc764 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:42:47.159112 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:47.159112 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:47.159112 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:47.159112 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:47.159112 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:47.159112 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:47.159112 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:47.159112 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:47.159112 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:47.159112 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:47.159112 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:47.159112 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:42:47.159112 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:42:47.159112 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:42:47.159112 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:42:47.160256 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:42:47.160256 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:42:47.160256 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:42:47.160256 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:42:47.160256 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:47.160256 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:47.160256 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:47.160256 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:47.160256 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:47.160256 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:47.160256 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:47.160256 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:47.160256 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:47.160256 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:47.160256 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:47.160256 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:47.161146 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:47.161146 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:47.161146 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:47.161146 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:47.161146 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:42:47.161146 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:42:47.161146 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:42:47.161146 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:42:47.161146 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:42:47.161146 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:42:47.161146 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:42:47.161146 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:47.165735 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 430/848] 10.222.0.52 () {56 vars in 1096 bytes} [Sun Sep 25 00:42:46 2022] GET /identity/v3/roles/admin => generated 83 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:42:47.167356 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6957f55e-db3c-4934-a20f-0ed5ed3fc764 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:47.181626 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c73f70f4-6884-4d83-86be-2d7af45367a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:47.182624 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73f70f4-6884-4d83-86be-2d7af45367a4 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:47.182850 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73f70f4-6884-4d83-86be-2d7af45367a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:47.183156 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73f70f4-6884-4d83-86be-2d7af45367a4 None admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:47.184093 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c73f70f4-6884-4d83-86be-2d7af45367a4 None admin] RBAC: Adding query filter params (name=service) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:47.184397 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c73f70f4-6884-4d83-86be-2d7af45367a4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:47.186647 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c73f70f4-6884-4d83-86be-2d7af45367a4 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:47.188788 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:47.202008 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 419/849] 10.222.0.52 () {56 vars in 1103 bytes} [Sun Sep 25 00:42:46 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:42:47.208374 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c73f70f4-6884-4d83-86be-2d7af45367a4 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:47.289469 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6957f55e-db3c-4934-a20f-0ed5ed3fc764 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:47.301775 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:47.327580 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689970ac-8c06-478b-a7bc-d9b84f411691 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:47.328199 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689970ac-8c06-478b-a7bc-d9b84f411691 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:47.328473 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689970ac-8c06-478b-a7bc-d9b84f411691 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:47.328735 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689970ac-8c06-478b-a7bc-d9b84f411691 None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:47.329718 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-689970ac-8c06-478b-a7bc-d9b84f411691 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:47.330022 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-689970ac-8c06-478b-a7bc-d9b84f411691 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:47.331393 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-689970ac-8c06-478b-a7bc-d9b84f411691 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:47.345659 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 431/850] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:42:47 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:42:47.347320 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689970ac-8c06-478b-a7bc-d9b84f411691 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:47.352948 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c73f70f4-6884-4d83-86be-2d7af45367a4 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:47.403666 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f829b09-468f-4ecd-9132-76f345dfd4cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:47.404319 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f829b09-468f-4ecd-9132-76f345dfd4cc None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:47.404601 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f829b09-468f-4ecd-9132-76f345dfd4cc None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:47.404945 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f829b09-468f-4ecd-9132-76f345dfd4cc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:47.419675 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f829b09-468f-4ecd-9132-76f345dfd4cc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:47.421183 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f829b09-468f-4ecd-9132-76f345dfd4cc None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:47.440860 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-6f829b09-468f-4ecd-9132-76f345dfd4cc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:47.440860 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:47.440860 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:47.440860 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:47.440860 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:47.440860 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:47.440860 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:47.440860 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:47.440860 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:47.440860 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:47.440860 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:47.440860 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:47.440860 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:47.440860 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:47.440860 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:47.440860 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:47.441890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:47.441890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:47.441890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:47.441890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:47.441890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:47.441890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:47.441890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:47.441890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:47.441890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:47.441890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:47.441890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:47.441890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:47.441890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:47.441890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:47.441890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:47.441890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:47.441890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:47.442948 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:47.442948 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:47.442948 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:47.442948 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 420/851] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:42:47.448833 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f829b09-468f-4ecd-9132-76f345dfd4cc None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:47.468396 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:47.493228 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689970ac-8c06-478b-a7bc-d9b84f411691 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:47.533534 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473e1700-d27a-4487-9a93-969f71b8226b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:47.534525 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473e1700-d27a-4487-9a93-969f71b8226b None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:47.535005 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473e1700-d27a-4487-9a93-969f71b8226b None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:47.535522 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473e1700-d27a-4487-9a93-969f71b8226b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:47.547267 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-473e1700-d27a-4487-9a93-969f71b8226b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:47.549089 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-473e1700-d27a-4487-9a93-969f71b8226b None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:47.565105 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-473e1700-d27a-4487-9a93-969f71b8226b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:47.565105 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:47.565105 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:47.565105 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:47.565105 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:47.565105 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:47.565105 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:47.565105 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:47.565105 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:47.565105 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:47.565105 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:47.565105 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:47.565105 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:47.565105 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:47.565105 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:47.565105 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:47.566147 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:47.566147 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:47.566147 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:47.566147 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:47.566147 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:47.566147 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:47.566147 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:47.566147 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:47.566147 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:47.566147 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:47.566147 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:47.566147 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:47.566147 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:47.566147 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:47.566147 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:47.566147 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:47.566147 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:47.567142 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:47.567142 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:47.567142 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:47.573914 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 432/852] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:47.575539 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473e1700-d27a-4487-9a93-969f71b8226b None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:47.583978 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f829b09-468f-4ecd-9132-76f345dfd4cc None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:47.603702 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:47.623948 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d7033db-70af-42fb-b80f-e0c6ea2d7e21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:47.624554 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7033db-70af-42fb-b80f-e0c6ea2d7e21 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:47.624823 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7033db-70af-42fb-b80f-e0c6ea2d7e21 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:47.625108 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7033db-70af-42fb-b80f-e0c6ea2d7e21 None admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:47.625855 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d7033db-70af-42fb-b80f-e0c6ea2d7e21 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:47.626155 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d7033db-70af-42fb-b80f-e0c6ea2d7e21 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:47.627503 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d7033db-70af-42fb-b80f-e0c6ea2d7e21 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:47.641586 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 421/853] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:42:47.647787 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d7033db-70af-42fb-b80f-e0c6ea2d7e21 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:47.690344 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473e1700-d27a-4487-9a93-969f71b8226b None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:47.721521 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:47.728725 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a165b30-5362-4cd0-a0e0-7f386e4accd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:47.729359 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a165b30-5362-4cd0-a0e0-7f386e4accd6 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:47.729634 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a165b30-5362-4cd0-a0e0-7f386e4accd6 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:47.729898 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a165b30-5362-4cd0-a0e0-7f386e4accd6 None admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:47.731086 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a165b30-5362-4cd0-a0e0-7f386e4accd6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:47.731086 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a165b30-5362-4cd0-a0e0-7f386e4accd6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:47.732288 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a165b30-5362-4cd0-a0e0-7f386e4accd6 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:47.746901 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 433/854] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:47.755091 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a165b30-5362-4cd0-a0e0-7f386e4accd6 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:47.766606 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d7033db-70af-42fb-b80f-e0c6ea2d7e21 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:47.811155 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b7a1350-4301-4b81-92ec-42f235b1337e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:47.812310 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b7a1350-4301-4b81-92ec-42f235b1337e None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:47.812791 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b7a1350-4301-4b81-92ec-42f235b1337e None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:47.813285 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b7a1350-4301-4b81-92ec-42f235b1337e None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:47.821950 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b7a1350-4301-4b81-92ec-42f235b1337e None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:47.823813 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b7a1350-4301-4b81-92ec-42f235b1337e None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:47.960089 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-9b7a1350-4301-4b81-92ec-42f235b1337e None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 25 00:42:47.960089 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:47.960089 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:47.960089 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:47.960089 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:47.960089 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:47.960089 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:47.960089 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:47.960089 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:47.960089 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:47.960089 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:47.960089 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:47.960089 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 00:42:47.960089 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 00:42:47.960089 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 00:42:47.960089 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 00:42:47.961116 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:47.961116 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:47.961116 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 00:42:47.961116 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:47.961116 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 00:42:47.961116 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:47.961116 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:47.961116 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:47.961116 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:47.961116 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:47.961116 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:47.961116 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:47.961116 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:47.961116 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:47.961116 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:47.961116 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:47.961116 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:47.961116 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:47.962146 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:47.962146 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:47.962146 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 00:42:47.962146 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 00:42:47.962146 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 00:42:47.962146 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 00:42:47.962146 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 00:42:47.962146 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 00:42:47.962146 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 25 00:42:47.962146 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:47.963305 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a165b30-5362-4cd0-a0e0-7f386e4accd6 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:47.964988 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 422/855] 10.222.0.52 () {56 vars in 1117 bytes} [Sun Sep 25 00:42:47 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 319 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:42:47.973522 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b7a1350-4301-4b81-92ec-42f235b1337e None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:48.018780 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:48.043119 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c793e8f-498e-488a-9347-504d9ded5086 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:48.044724 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c793e8f-498e-488a-9347-504d9ded5086 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:48.045451 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c793e8f-498e-488a-9347-504d9ded5086 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:48.046177 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c793e8f-498e-488a-9347-504d9ded5086 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:48.062090 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c793e8f-498e-488a-9347-504d9ded5086 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:48.065846 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c793e8f-498e-488a-9347-504d9ded5086 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:48.086709 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b7a1350-4301-4b81-92ec-42f235b1337e None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:48.094130 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-0c793e8f-498e-488a-9347-504d9ded5086 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 25 00:42:48.094130 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:48.094130 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:48.094130 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:48.094130 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:48.094130 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:48.094130 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:48.094130 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:48.094130 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:48.094130 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:48.094130 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:48.094130 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:48.094130 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 00:42:48.094130 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 00:42:48.094130 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 00:42:48.094130 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 00:42:48.095940 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:48.095940 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:48.095940 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 00:42:48.095940 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:48.095940 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 00:42:48.095940 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:48.095940 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:48.095940 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:48.095940 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:48.095940 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:48.095940 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:48.095940 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:48.095940 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:48.095940 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:48.095940 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:48.095940 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:48.095940 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:48.095940 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:48.097964 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:48.097964 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:48.097964 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 00:42:48.097964 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 00:42:48.097964 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 00:42:48.097964 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 00:42:48.097964 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 00:42:48.097964 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 00:42:48.097964 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 25 00:42:48.097964 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:48.109563 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 434/856] 10.222.0.52 () {56 vars in 1096 bytes} [Sun Sep 25 00:42:47 2022] GET /identity/v3/users/swift => generated 83 bytes in 349 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:42:48.111510 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c793e8f-498e-488a-9347-504d9ded5086 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:48.127170 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd5b5685-3ec4-43a5-87fd-00dddcf1d743 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:48.127940 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5b5685-3ec4-43a5-87fd-00dddcf1d743 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:48.128262 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5b5685-3ec4-43a5-87fd-00dddcf1d743 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:48.128680 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5b5685-3ec4-43a5-87fd-00dddcf1d743 None admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:48.129551 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd5b5685-3ec4-43a5-87fd-00dddcf1d743 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:48.129915 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd5b5685-3ec4-43a5-87fd-00dddcf1d743 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:48.131953 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd5b5685-3ec4-43a5-87fd-00dddcf1d743 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:48.154670 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 423/857] 10.222.0.52 () {56 vars in 1149 bytes} [Sun Sep 25 00:42:47 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:48.164682 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd5b5685-3ec4-43a5-87fd-00dddcf1d743 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:48.220327 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:48.274359 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:48.282023 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd5b5685-3ec4-43a5-87fd-00dddcf1d743 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:48.308813 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c793e8f-498e-488a-9347-504d9ded5086 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:48.319573 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dbe83d2-89b5-4fc5-996f-0920b5a773af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:48.320154 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbe83d2-89b5-4fc5-996f-0920b5a773af None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:48.320481 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbe83d2-89b5-4fc5-996f-0920b5a773af None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:48.320768 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbe83d2-89b5-4fc5-996f-0920b5a773af None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:48.372333 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1513384-59ab-4f89-b520-53e3b6a1cf24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:48.373938 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1513384-59ab-4f89-b520-53e3b6a1cf24 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:48.374695 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1513384-59ab-4f89-b520-53e3b6a1cf24 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:48.375424 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1513384-59ab-4f89-b520-53e3b6a1cf24 None admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:48.377036 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1513384-59ab-4f89-b520-53e3b6a1cf24 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:48.377849 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1513384-59ab-4f89-b520-53e3b6a1cf24 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:48.381152 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1513384-59ab-4f89-b520-53e3b6a1cf24 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:48.397545 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dbe83d2-89b5-4fc5-996f-0920b5a773af None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:48.399133 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dbe83d2-89b5-4fc5-996f-0920b5a773af None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:48.421705 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 435/858] 10.222.0.52 () {56 vars in 1135 bytes} [Sun Sep 25 00:42:48 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 311 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:48.426370 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1513384-59ab-4f89-b520-53e3b6a1cf24 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:48.503176 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:48.512607 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 424/859] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:42:48 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 348 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 25 00:42:48.522192 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dbe83d2-89b5-4fc5-996f-0920b5a773af None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:48.593043 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1513384-59ab-4f89-b520-53e3b6a1cf24 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:48.606681 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:48.630821 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dbe83d2-89b5-4fc5-996f-0920b5a773af None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:48.638283 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa9a6941-1180-428b-9b15-ceed336ced43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:48.639473 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa9a6941-1180-428b-9b15-ceed336ced43 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:48.639959 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa9a6941-1180-428b-9b15-ceed336ced43 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:48.640544 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa9a6941-1180-428b-9b15-ceed336ced43 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:48.670917 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be67fd78-5ed9-41fa-aee0-47d409205e7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:48.671533 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be67fd78-5ed9-41fa-aee0-47d409205e7e None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:48.671806 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be67fd78-5ed9-41fa-aee0-47d409205e7e None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:48.672080 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be67fd78-5ed9-41fa-aee0-47d409205e7e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:48.683153 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be67fd78-5ed9-41fa-aee0-47d409205e7e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:48.684610 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be67fd78-5ed9-41fa-aee0-47d409205e7e None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:48.701741 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-be67fd78-5ed9-41fa-aee0-47d409205e7e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:48.701741 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:48.701741 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:48.701741 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:48.701741 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:48.701741 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:48.701741 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:48.701741 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:48.701741 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:48.701741 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:48.701741 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:48.701741 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:48.701741 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:48.701741 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:48.701741 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:48.701741 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:48.703478 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:48.703478 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:48.703478 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:48.703478 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:48.703478 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:48.703478 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:48.703478 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:48.703478 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:48.703478 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:48.703478 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:48.703478 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:48.703478 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:48.703478 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:48.703478 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:48.703478 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:48.703478 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:48.703478 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:48.704783 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:48.704783 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:48.704783 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:48.707576 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 425/860] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:48.708485 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be67fd78-5ed9-41fa-aee0-47d409205e7e None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:48.744778 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa9a6941-1180-428b-9b15-ceed336ced43 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:48.746858 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa9a6941-1180-428b-9b15-ceed336ced43 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:48.761229 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:48.816154 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be67fd78-5ed9-41fa-aee0-47d409205e7e None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:48.852243 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad27ab0-0f9a-4b21-93ae-194d0b7ec722 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:48.852765 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad27ab0-0f9a-4b21-93ae-194d0b7ec722 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:48.853037 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad27ab0-0f9a-4b21-93ae-194d0b7ec722 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:48.853319 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad27ab0-0f9a-4b21-93ae-194d0b7ec722 None admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:48.853958 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ad27ab0-0f9a-4b21-93ae-194d0b7ec722 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:48.854260 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ad27ab0-0f9a-4b21-93ae-194d0b7ec722 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:48.855620 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ad27ab0-0f9a-4b21-93ae-194d0b7ec722 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:48.869231 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 436/861] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:42:48 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 444 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 25 00:42:48.871449 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa9a6941-1180-428b-9b15-ceed336ced43 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:48.877621 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 426/862] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:48.878607 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad27ab0-0f9a-4b21-93ae-194d0b7ec722 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:48.885434 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:48.991791 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad27ab0-0f9a-4b21-93ae-194d0b7ec722 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:48.994093 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa9a6941-1180-428b-9b15-ceed336ced43 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:49.003276 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:49.033026 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb1006f-1edd-4259-b346-9c9742e913b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:49.033690 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb1006f-1edd-4259-b346-9c9742e913b5 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:49.033959 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb1006f-1edd-4259-b346-9c9742e913b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:49.034222 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb1006f-1edd-4259-b346-9c9742e913b5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:49.038602 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-189fbf84-7a7d-45c7-8e98-701d78daa4d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:49.039500 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189fbf84-7a7d-45c7-8e98-701d78daa4d1 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:49.040170 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189fbf84-7a7d-45c7-8e98-701d78daa4d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:49.040681 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189fbf84-7a7d-45c7-8e98-701d78daa4d1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:49.043004 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecb1006f-1edd-4259-b346-9c9742e913b5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:49.044584 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecb1006f-1edd-4259-b346-9c9742e913b5 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:49.054013 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-189fbf84-7a7d-45c7-8e98-701d78daa4d1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:49.056004 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-189fbf84-7a7d-45c7-8e98-701d78daa4d1 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:49.057032 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-ecb1006f-1edd-4259-b346-9c9742e913b5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:42:49.057032 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:49.057032 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:49.057032 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:49.057032 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:49.057032 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:49.057032 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:49.057032 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:49.057032 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:49.057032 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:49.057032 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:49.057032 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:49.057032 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 00:42:49.057032 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 00:42:49.057032 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 00:42:49.058112 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 00:42:49.058112 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:49.058112 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:49.058112 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:49.058112 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:49.058112 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:49.058112 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:49.058112 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:49.058112 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:49.058112 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:49.058112 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:49.058112 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:49.058112 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:49.058112 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:49.058112 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:49.058112 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:49.058112 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:49.059277 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 00:42:49.059277 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 00:42:49.059277 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 00:42:49.059277 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 00:42:49.059277 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 00:42:49.059277 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 00:42:49.059277 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:42:49.059277 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:49.063014 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 427/863] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:42:48 2022] GET /identity/v3/projects/service => generated 88 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:49.063921 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb1006f-1edd-4259-b346-9c9742e913b5 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:49.077530 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-189fbf84-7a7d-45c7-8e98-701d78daa4d1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:49.077530 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:49.077530 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:49.077530 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:49.077530 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:49.077530 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:49.077530 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:49.077530 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:49.077530 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:49.077530 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:49.077530 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:49.077530 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:49.077530 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:49.077530 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:49.077530 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:49.077530 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:49.078692 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:49.078692 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:49.078692 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:49.078692 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:49.078692 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:49.078692 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:49.078692 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:49.078692 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:49.078692 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:49.078692 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:49.078692 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:49.078692 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:49.078692 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:49.078692 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:49.078692 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:49.078692 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:49.078692 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:49.079794 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:49.079794 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:49.079794 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:49.081561 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 437/864] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:49.083079 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b24bb1-1cf8-4a90-bf94-02237f757d26 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:49.083521 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b24bb1-1cf8-4a90-bf94-02237f757d26 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:49.084569 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b24bb1-1cf8-4a90-bf94-02237f757d26 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:49.086292 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 438/865] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:42:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:42:49.092019 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b24bb1-1cf8-4a90-bf94-02237f757d26 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:49.158781 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:49.171702 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb1006f-1edd-4259-b346-9c9742e913b5 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:49.200113 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b24bb1-1cf8-4a90-bf94-02237f757d26 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:49.206740 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c51812-e56e-40c5-8c06-d5ee6eed5507 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:49.207343 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c51812-e56e-40c5-8c06-d5ee6eed5507 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:49.207618 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c51812-e56e-40c5-8c06-d5ee6eed5507 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:49.207878 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c51812-e56e-40c5-8c06-d5ee6eed5507 None admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:49.208557 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6c51812-e56e-40c5-8c06-d5ee6eed5507 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:49.208890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6c51812-e56e-40c5-8c06-d5ee6eed5507 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:49.210561 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6c51812-e56e-40c5-8c06-d5ee6eed5507 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:49.223948 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 428/866] 10.222.0.52 () {56 vars in 1148 bytes} [Sun Sep 25 00:42:49 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:49.225862 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7506278-aa9c-4ed6-992e-e668accec47f None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:49.226152 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7506278-aa9c-4ed6-992e-e668accec47f None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:49.226419 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7506278-aa9c-4ed6-992e-e668accec47f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:49.238638 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15341e2a-8fac-4563-b607-3200af7c5c76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:49.239237 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15341e2a-8fac-4563-b607-3200af7c5c76 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:49.239503 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15341e2a-8fac-4563-b607-3200af7c5c76 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:49.239763 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15341e2a-8fac-4563-b607-3200af7c5c76 None admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:49.241009 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15341e2a-8fac-4563-b607-3200af7c5c76 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:49.241324 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15341e2a-8fac-4563-b607-3200af7c5c76 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:49.242767 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15341e2a-8fac-4563-b607-3200af7c5c76 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:49.244915 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-e7506278-aa9c-4ed6-992e-e668accec47f None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:42:49.246146 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-e7506278-aa9c-4ed6-992e-e668accec47f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:42:49.254654 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 439/867] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:49.255899 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce0375d-3a8b-4998-bfd0-8e4760aa3720 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:49.256217 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce0375d-3a8b-4998-bfd0-8e4760aa3720 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:49.256482 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce0375d-3a8b-4998-bfd0-8e4760aa3720 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:49.257729 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 440/868] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:42:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:42:49.261889 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce0375d-3a8b-4998-bfd0-8e4760aa3720 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:49.273918 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:49.319850 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 429/869] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:42:49 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:42:49.320550 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7506278-aa9c-4ed6-992e-e668accec47f None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:49.369501 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce0375d-3a8b-4998-bfd0-8e4760aa3720 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:49.396102 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:49.409964 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c50a975-aa3f-43c9-a1dc-ccca102835d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:49.410486 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c50a975-aa3f-43c9-a1dc-ccca102835d3 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:49.410763 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c50a975-aa3f-43c9-a1dc-ccca102835d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:49.411035 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c50a975-aa3f-43c9-a1dc-ccca102835d3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:49.411792 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c50a975-aa3f-43c9-a1dc-ccca102835d3 None admin] RBAC: Adding query filter params (role.id=a8e34e460e8a4ed7bcaa54da3c2d5a0c, scope.project.id=9db0fd07fc494203b2ada13f3cd046d3, user.id=3f20ffd8df994c1db1b1e7248db5b5a3) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:49.412045 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c50a975-aa3f-43c9-a1dc-ccca102835d3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:49.413731 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c50a975-aa3f-43c9-a1dc-ccca102835d3 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:49.420054 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 441/870] 10.222.0.52 () {56 vars in 1374 bytes} [Sun Sep 25 00:42:49 2022] GET /identity/v3/role_assignments?user.id=3f20ffd8df994c1db1b1e7248db5b5a3&scope.project.id=9db0fd07fc494203b2ada13f3cd046d3&role.id=a8e34e460e8a4ed7bcaa54da3c2d5a0c => generated 608 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:49.421224 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123f4f55-86dd-455d-808b-599c96ecd298 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:49.421535 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123f4f55-86dd-455d-808b-599c96ecd298 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:49.421804 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123f4f55-86dd-455d-808b-599c96ecd298 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:49.441762 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-123f4f55-86dd-455d-808b-599c96ecd298 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:42:49.443133 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-123f4f55-86dd-455d-808b-599c96ecd298 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:42:49.465081 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7506278-aa9c-4ed6-992e-e668accec47f None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:49.526765 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3879c847-ec96-4723-b9aa-6c2f12385c20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:49.527507 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 442/871] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:42:49 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:42:49.527970 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3879c847-ec96-4723-b9aa-6c2f12385c20 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:49.528348 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3879c847-ec96-4723-b9aa-6c2f12385c20 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:49.528627 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3879c847-ec96-4723-b9aa-6c2f12385c20 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:49.542110 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3879c847-ec96-4723-b9aa-6c2f12385c20 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:49.544156 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3879c847-ec96-4723-b9aa-6c2f12385c20 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:49.558708 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-3879c847-ec96-4723-b9aa-6c2f12385c20 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:42:49.558708 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:49.558708 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:49.558708 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:49.558708 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:49.558708 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:49.558708 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:49.558708 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:49.558708 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:49.558708 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:49.558708 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:49.558708 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:49.558708 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 00:42:49.558708 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 00:42:49.558708 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 00:42:49.559725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 00:42:49.559725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:49.559725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:49.559725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:49.559725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:49.559725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:49.559725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:49.559725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:49.559725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:49.559725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:49.559725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:49.559725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:49.559725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:49.559725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:49.559725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:49.559725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:49.559725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:49.560692 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 00:42:49.560692 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 00:42:49.560692 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 00:42:49.560692 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 00:42:49.560692 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 00:42:49.560692 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 00:42:49.560692 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:42:49.560692 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:49.568851 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 430/872] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:42:49 2022] GET /identity/v3/projects/service => generated 88 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:49.571634 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-123f4f55-86dd-455d-808b-599c96ecd298 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:49.595560 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3879c847-ec96-4723-b9aa-6c2f12385c20 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:49.622206 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:49.718478 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3879c847-ec96-4723-b9aa-6c2f12385c20 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:49.728433 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-123f4f55-86dd-455d-808b-599c96ecd298 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:49.750152 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:49.760267 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba973e9-202d-4721-9967-b3b3b702aea1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:49.761058 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba973e9-202d-4721-9967-b3b3b702aea1 None admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:49.761172 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba973e9-202d-4721-9967-b3b3b702aea1 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:49.761426 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba973e9-202d-4721-9967-b3b3b702aea1 None admin] PATH_INFO: `/v3/services` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:49.762081 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba973e9-202d-4721-9967-b3b3b702aea1 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:49.763436 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba973e9-202d-4721-9967-b3b3b702aea1 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:49.777791 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae2bcdad-8c57-4834-a34f-f95814bdd87a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:49.779085 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2bcdad-8c57-4834-a34f-f95814bdd87a None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:49.779794 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2bcdad-8c57-4834-a34f-f95814bdd87a None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:49.780170 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2bcdad-8c57-4834-a34f-f95814bdd87a None admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:49.780879 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae2bcdad-8c57-4834-a34f-f95814bdd87a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:49.781358 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae2bcdad-8c57-4834-a34f-f95814bdd87a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:49.782744 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae2bcdad-8c57-4834-a34f-f95814bdd87a None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:49.785404 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 431/873] 10.222.0.52 () {60 vars in 1141 bytes} [Sun Sep 25 00:42:49 2022] POST /identity/v3/services => generated 251 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:42:49.800906 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 443/874] 10.222.0.52 () {56 vars in 1148 bytes} [Sun Sep 25 00:42:49 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:49.806538 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba973e9-202d-4721-9967-b3b3b702aea1 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:49.808923 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae2bcdad-8c57-4834-a34f-f95814bdd87a None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:49.913757 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:49.949581 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba973e9-202d-4721-9967-b3b3b702aea1 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:49.965157 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae2bcdad-8c57-4834-a34f-f95814bdd87a None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:50.001700 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff69d55-bbb3-4be9-9f33-4ba32649fbdc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:50.002873 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff69d55-bbb3-4be9-9f33-4ba32649fbdc None admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:50.003357 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff69d55-bbb3-4be9-9f33-4ba32649fbdc None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:50.003872 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff69d55-bbb3-4be9-9f33-4ba32649fbdc None admin] PATH_INFO: `/v3/services` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:50.005327 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff69d55-bbb3-4be9-9f33-4ba32649fbdc None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:50.007750 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff69d55-bbb3-4be9-9f33-4ba32649fbdc None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:50.016348 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3b9e43a-257c-47cd-81c6-1cfac8ad10db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:50.017071 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b9e43a-257c-47cd-81c6-1cfac8ad10db None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:50.017422 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b9e43a-257c-47cd-81c6-1cfac8ad10db None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:50.017694 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b9e43a-257c-47cd-81c6-1cfac8ad10db None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:50.018478 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3b9e43a-257c-47cd-81c6-1cfac8ad10db None admin] RBAC: Adding query filter params (role.id=2c242abf6ed8410ab0a4a354ca1794fb, scope.project.id=9db0fd07fc494203b2ada13f3cd046d3, user.id=da606b3906dd4171bd76fd281e03e258) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:50.018783 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3b9e43a-257c-47cd-81c6-1cfac8ad10db None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:50.021447 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3b9e43a-257c-47cd-81c6-1cfac8ad10db None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:50.028131 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 432/875] 10.222.0.52 () {60 vars in 1141 bytes} [Sun Sep 25 00:42:49 2022] POST /identity/v3/services => generated 269 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:42:50.033309 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 444/876] 10.222.0.52 () {56 vars in 1374 bytes} [Sun Sep 25 00:42:49 2022] GET /identity/v3/role_assignments?user.id=da606b3906dd4171bd76fd281e03e258&scope.project.id=9db0fd07fc494203b2ada13f3cd046d3&role.id=2c242abf6ed8410ab0a4a354ca1794fb => generated 608 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:51.890235 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51b16bd-6f89-40ae-a795-a94f0c50e523 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:51.890733 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51b16bd-6f89-40ae-a795-a94f0c50e523 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:51.890733 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51b16bd-6f89-40ae-a795-a94f0c50e523 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:51.892308 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 433/877] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:42:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:42:51.898926 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60483553-44e0-441e-a5a3-a9f097b7a8b4 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:51.899897 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60483553-44e0-441e-a5a3-a9f097b7a8b4 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:51.900376 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60483553-44e0-441e-a5a3-a9f097b7a8b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:51.927611 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-60483553-44e0-441e-a5a3-a9f097b7a8b4 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:42:51.930730 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-60483553-44e0-441e-a5a3-a9f097b7a8b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:42:52.001040 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:52.064886 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 445/878] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:42:51 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:42:52.101808 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9365dcb-5dbf-4b0f-a811-8f707416cfc6 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:52.102092 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9365dcb-5dbf-4b0f-a811-8f707416cfc6 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:52.102367 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9365dcb-5dbf-4b0f-a811-8f707416cfc6 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:52.103968 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 434/879] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:42:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:42:52.109773 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598cdc33-b8c2-4ff3-81c6-1183127f2a91 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:52.110052 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598cdc33-b8c2-4ff3-81c6-1183127f2a91 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:52.110319 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598cdc33-b8c2-4ff3-81c6-1183127f2a91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:52.130181 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-598cdc33-b8c2-4ff3-81c6-1183127f2a91 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:42:52.131807 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-598cdc33-b8c2-4ff3-81c6-1183127f2a91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:42:52.209542 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 446/880] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:42:52 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:42:52.344000 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9365dcb-5dbf-4b0f-a811-8f707416cfc6 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:52.351994 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9a8276-a901-42df-b073-7f132973a786 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:52.352304 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9a8276-a901-42df-b073-7f132973a786 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:52.352548 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9a8276-a901-42df-b073-7f132973a786 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:52.354231 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 447/881] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:42:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:42:52.361537 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc554f2-ced7-451c-9d1b-f194062a709d None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:52.362450 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc554f2-ced7-451c-9d1b-f194062a709d None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:52.362852 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc554f2-ced7-451c-9d1b-f194062a709d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:52.385959 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-edc554f2-ced7-451c-9d1b-f194062a709d None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:42:52.387256 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-edc554f2-ced7-451c-9d1b-f194062a709d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:42:52.461333 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 448/882] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:42:52 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 25 00:42:52.462707 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e30843c-58a1-4ad7-87e8-8f6df51bbe9a None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:52.463017 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e30843c-58a1-4ad7-87e8-8f6df51bbe9a None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:52.463286 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e30843c-58a1-4ad7-87e8-8f6df51bbe9a None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:52.468802 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 449/883] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:42:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:42:52.470387 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae4dd1a-cbc0-4a12-a1e4-83b2a862edf4 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:52.470662 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae4dd1a-cbc0-4a12-a1e4-83b2a862edf4 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:52.470928 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae4dd1a-cbc0-4a12-a1e4-83b2a862edf4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:52.483103 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9365dcb-5dbf-4b0f-a811-8f707416cfc6 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:52.491617 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-bae4dd1a-cbc0-4a12-a1e4-83b2a862edf4 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:42:52.492782 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-bae4dd1a-cbc0-4a12-a1e4-83b2a862edf4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:42:52.515760 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:52.527447 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2227b797-e8ea-43ae-b6a6-1c8b1beb1415 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:52.528144 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2227b797-e8ea-43ae-b6a6-1c8b1beb1415 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:52.528426 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2227b797-e8ea-43ae-b6a6-1c8b1beb1415 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:52.528696 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2227b797-e8ea-43ae-b6a6-1c8b1beb1415 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:52.530868 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:52.544119 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2227b797-e8ea-43ae-b6a6-1c8b1beb1415 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:42:52.549343 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-2227b797-e8ea-43ae-b6a6-1c8b1beb1415 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 25 00:42:52.549343 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:52.549343 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:52.549343 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:52.549343 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:52.549343 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:52.549343 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:52.549343 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:52.549343 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:52.549343 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:52.549343 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:52.549343 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:52.549343 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:42:52.549343 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:42:52.549343 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:42:52.549343 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:42:52.550451 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:42:52.550451 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:42:52.550451 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:42:52.550451 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:42:52.550451 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:52.550451 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:52.550451 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:52.550451 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:52.550451 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:52.550451 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:52.550451 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:52.550451 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:52.550451 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:52.550451 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:52.550451 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:52.550451 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:52.551413 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:52.551413 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:52.551413 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:52.551413 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:52.551413 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:42:52.551413 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:42:52.551413 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:42:52.551413 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:42:52.551413 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:42:52.551413 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:42:52.551413 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 25 00:42:52.551413 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:52.554117 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 435/884] 10.222.0.52 () {56 vars in 1120 bytes} [Sun Sep 25 00:42:52 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:52.555208 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2227b797-e8ea-43ae-b6a6-1c8b1beb1415 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:52.572308 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 450/885] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:42:52 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:42:52.573537 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae4dd1a-cbc0-4a12-a1e4-83b2a862edf4 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:52.688681 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae4dd1a-cbc0-4a12-a1e4-83b2a862edf4 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:52.710078 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2227b797-e8ea-43ae-b6a6-1c8b1beb1415 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:52.731548 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe76f73d-3540-4dba-af64-32568482d47f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:52.732654 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe76f73d-3540-4dba-af64-32568482d47f None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:52.733139 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe76f73d-3540-4dba-af64-32568482d47f None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:52.733751 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe76f73d-3540-4dba-af64-32568482d47f None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:52.734696 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe76f73d-3540-4dba-af64-32568482d47f None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:52.735222 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe76f73d-3540-4dba-af64-32568482d47f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:52.737028 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe76f73d-3540-4dba-af64-32568482d47f None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:52.747631 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 451/886] 10.222.0.52 () {56 vars in 1115 bytes} [Sun Sep 25 00:42:52 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:52.748875 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe76f73d-3540-4dba-af64-32568482d47f None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:52.755694 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f01572a-40d6-4e5c-86d7-c5437a5d26de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:52.756907 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f01572a-40d6-4e5c-86d7-c5437a5d26de None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:52.757187 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f01572a-40d6-4e5c-86d7-c5437a5d26de None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:52.757503 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f01572a-40d6-4e5c-86d7-c5437a5d26de None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:52.758094 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f01572a-40d6-4e5c-86d7-c5437a5d26de None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:52.760203 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f01572a-40d6-4e5c-86d7-c5437a5d26de None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:52.770969 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-9f01572a-40d6-4e5c-86d7-c5437a5d26de None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 25 00:42:52.770969 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:52.770969 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 00:42:52.770969 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 00:42:52.770969 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 00:42:52.770969 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 00:42:52.770969 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 00:42:52.770969 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:42:52.770969 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 25 00:42:52.770969 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:52.770969 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 00:42:52.770969 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:52.770969 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:52.770969 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:52.770969 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:52.770969 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:52.770969 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:52.772703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:52.772703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:52.772703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:52.772703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:52.772703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:52.772703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:52.772703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 00:42:52.772703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 00:42:52.772703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 00:42:52.772703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 00:42:52.772703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:52.772703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:52.772703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:52.772703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:52.772703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:52.772703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:52.773652 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:52.773652 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:52.773652 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:52.773652 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:52.773652 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:52.773652 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:52.773652 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:52.773652 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:52.773652 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:52.773652 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:52.773652 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 00:42:52.773652 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:42:52.773652 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 25 00:42:52.773652 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:52.773652 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 436/887] 10.222.0.52 () {56 vars in 1114 bytes} [Sun Sep 25 00:42:52 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:52.775674 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f01572a-40d6-4e5c-86d7-c5437a5d26de None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:52.776311 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:52.845494 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:52.875624 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe76f73d-3540-4dba-af64-32568482d47f None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:52.893993 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f01572a-40d6-4e5c-86d7-c5437a5d26de None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:52.911498 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bafadb1-d6c4-405e-b191-bf7e65ab9d0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:52.912703 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bafadb1-d6c4-405e-b191-bf7e65ab9d0c None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:52.912980 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bafadb1-d6c4-405e-b191-bf7e65ab9d0c None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:52.913271 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bafadb1-d6c4-405e-b191-bf7e65ab9d0c None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:52.914060 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bafadb1-d6c4-405e-b191-bf7e65ab9d0c None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:52.915432 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bafadb1-d6c4-405e-b191-bf7e65ab9d0c None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:52.927069 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-2bafadb1-d6c4-405e-b191-bf7e65ab9d0c None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 25 00:42:52.927069 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:52.927069 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 00:42:52.927069 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 00:42:52.927069 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 00:42:52.927069 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 00:42:52.927069 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 00:42:52.927069 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:42:52.927069 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 25 00:42:52.927069 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:52.927069 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 00:42:52.927069 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:52.927069 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:52.927069 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:52.927069 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:52.927069 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:52.929097 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:52.929097 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:52.929097 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:52.929097 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:52.929097 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:52.929097 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:52.929097 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:52.929097 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 00:42:52.929097 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 00:42:52.929097 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 00:42:52.929097 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 00:42:52.929097 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:52.929097 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:52.929097 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:52.929097 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:52.929097 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:52.929097 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:52.930398 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:52.930398 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:52.930398 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:52.930398 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:52.930398 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:52.930398 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:52.930398 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:52.930398 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:52.930398 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:52.930398 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:52.930398 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 00:42:52.930398 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:42:52.930398 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 25 00:42:52.930398 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:52.930398 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 452/888] 10.222.0.52 () {56 vars in 1132 bytes} [Sun Sep 25 00:42:52 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:52.931395 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bafadb1-d6c4-405e-b191-bf7e65ab9d0c None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:52.936169 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e72b6f26-d1ea-4079-9d31-ac890d80cd56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:52.936808 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72b6f26-d1ea-4079-9d31-ac890d80cd56 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:52.937079 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72b6f26-d1ea-4079-9d31-ac890d80cd56 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:52.937375 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72b6f26-d1ea-4079-9d31-ac890d80cd56 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:52.948868 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e72b6f26-d1ea-4079-9d31-ac890d80cd56 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:52.950333 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e72b6f26-d1ea-4079-9d31-ac890d80cd56 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:52.966220 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-e72b6f26-d1ea-4079-9d31-ac890d80cd56 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:52.966220 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:52.966220 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:52.966220 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:52.966220 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:52.966220 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:52.966220 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:52.966220 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:52.966220 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:52.966220 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:52.966220 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:52.966220 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:52.966220 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:52.966220 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:52.966220 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:52.966220 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:52.967440 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:52.967440 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:52.967440 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:52.967440 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:52.967440 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:52.967440 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:52.967440 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:52.967440 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:52.967440 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:52.967440 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:52.967440 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:52.967440 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:52.967440 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:52.967440 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:52.967440 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:52.967440 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:52.967440 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:52.968837 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:52.968837 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:52.968837 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:52.968837 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 437/889] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:52.968837 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e72b6f26-d1ea-4079-9d31-ac890d80cd56 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:53.019286 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:53.038991 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bafadb1-d6c4-405e-b191-bf7e65ab9d0c None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:53.070308 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e72b6f26-d1ea-4079-9d31-ac890d80cd56 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:53.074130 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c5ea68-e8d5-420a-ae2f-5fed73afbdc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:53.074744 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c5ea68-e8d5-420a-ae2f-5fed73afbdc9 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:53.075023 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c5ea68-e8d5-420a-ae2f-5fed73afbdc9 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:53.075293 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c5ea68-e8d5-420a-ae2f-5fed73afbdc9 None admin] PATH_INFO: `/v3/services` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:53.075926 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5c5ea68-e8d5-420a-ae2f-5fed73afbdc9 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:53.076229 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5c5ea68-e8d5-420a-ae2f-5fed73afbdc9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:53.077647 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5c5ea68-e8d5-420a-ae2f-5fed73afbdc9 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:53.081746 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:53.082653 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 453/890] 10.222.0.52 () {56 vars in 1114 bytes} [Sun Sep 25 00:42:52 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:53.083302 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c5ea68-e8d5-420a-ae2f-5fed73afbdc9 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:53.109439 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2c8a71-dee2-4f44-88b9-b09cf153482b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:53.110269 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2c8a71-dee2-4f44-88b9-b09cf153482b None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:53.110653 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2c8a71-dee2-4f44-88b9-b09cf153482b None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:53.111000 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2c8a71-dee2-4f44-88b9-b09cf153482b None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:53.111583 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d2c8a71-dee2-4f44-88b9-b09cf153482b None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:53.113209 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d2c8a71-dee2-4f44-88b9-b09cf153482b None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:53.122906 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-9d2c8a71-dee2-4f44-88b9-b09cf153482b None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 25 00:42:53.122906 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:53.122906 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 00:42:53.122906 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 00:42:53.122906 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 00:42:53.122906 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 00:42:53.122906 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 00:42:53.122906 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:42:53.122906 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 25 00:42:53.122906 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:53.122906 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 00:42:53.122906 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:53.122906 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:53.122906 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:53.122906 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:53.122906 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:53.122906 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:53.124779 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:53.124779 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:53.124779 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:53.124779 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:53.124779 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:53.124779 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:53.124779 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 00:42:53.124779 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 00:42:53.124779 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 00:42:53.124779 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 00:42:53.124779 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:53.124779 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:53.124779 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:53.124779 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:53.124779 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:53.124779 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:53.126656 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:53.126656 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:53.126656 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:53.126656 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:53.126656 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:53.126656 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:53.126656 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:53.126656 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:53.126656 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:53.126656 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:53.126656 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 00:42:53.126656 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:42:53.126656 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 25 00:42:53.126656 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:53.126656 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 438/891] 10.222.0.52 () {56 vars in 1126 bytes} [Sun Sep 25 00:42:52 2022] GET /identity/v3/services/object-store => generated 93 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:53.128429 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2c8a71-dee2-4f44-88b9-b09cf153482b None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:53.196384 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c5ea68-e8d5-420a-ae2f-5fed73afbdc9 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:53.231690 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c542b5-87c4-4374-9851-1953446454a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:53.232120 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2c8a71-dee2-4f44-88b9-b09cf153482b None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:53.232296 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c542b5-87c4-4374-9851-1953446454a7 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:53.232572 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c542b5-87c4-4374-9851-1953446454a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:53.232837 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c542b5-87c4-4374-9851-1953446454a7 None admin] PATH_INFO: `/v3/services` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:53.233490 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c542b5-87c4-4374-9851-1953446454a7 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:53.233792 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c542b5-87c4-4374-9851-1953446454a7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:53.235149 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c542b5-87c4-4374-9851-1953446454a7 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:53.240075 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 454/892] 10.222.0.52 () {56 vars in 1126 bytes} [Sun Sep 25 00:42:53 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:53.240947 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c542b5-87c4-4374-9851-1953446454a7 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:53.253674 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:53.268726 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b8d54f-78bc-4d03-9f2f-84818be5399a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:53.269424 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b8d54f-78bc-4d03-9f2f-84818be5399a None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:53.269644 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b8d54f-78bc-4d03-9f2f-84818be5399a None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:53.269910 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b8d54f-78bc-4d03-9f2f-84818be5399a None admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:53.270553 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72b8d54f-78bc-4d03-9f2f-84818be5399a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:53.270849 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72b8d54f-78bc-4d03-9f2f-84818be5399a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:53.273114 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72b8d54f-78bc-4d03-9f2f-84818be5399a None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:53.284614 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 439/893] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:53.286169 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b8d54f-78bc-4d03-9f2f-84818be5399a None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:53.322537 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:53.346226 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c542b5-87c4-4374-9851-1953446454a7 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:53.381883 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2479afb3-2de0-488a-a0db-35c8c5db6a08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:53.382426 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2479afb3-2de0-488a-a0db-35c8c5db6a08 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:53.382702 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2479afb3-2de0-488a-a0db-35c8c5db6a08 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:53.382969 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2479afb3-2de0-488a-a0db-35c8c5db6a08 None admin] PATH_INFO: `/v3/services` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:53.383645 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2479afb3-2de0-488a-a0db-35c8c5db6a08 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:53.383948 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2479afb3-2de0-488a-a0db-35c8c5db6a08 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:53.385332 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2479afb3-2de0-488a-a0db-35c8c5db6a08 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:53.390271 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 455/894] 10.222.0.52 () {56 vars in 1114 bytes} [Sun Sep 25 00:42:53 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:53.391067 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2479afb3-2de0-488a-a0db-35c8c5db6a08 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:53.391182 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b8d54f-78bc-4d03-9f2f-84818be5399a None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:53.426671 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f888995-db08-4fec-a7b6-b1f2fbfcdb2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:53.427363 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f888995-db08-4fec-a7b6-b1f2fbfcdb2f None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:53.427642 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f888995-db08-4fec-a7b6-b1f2fbfcdb2f None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:53.428405 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f888995-db08-4fec-a7b6-b1f2fbfcdb2f None admin] PATH_INFO: `/v3/services` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:53.429030 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f888995-db08-4fec-a7b6-b1f2fbfcdb2f None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:53.429371 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f888995-db08-4fec-a7b6-b1f2fbfcdb2f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:53.430744 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f888995-db08-4fec-a7b6-b1f2fbfcdb2f None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:53.435469 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 440/895] 10.222.0.52 () {56 vars in 1122 bytes} [Sun Sep 25 00:42:53 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:53.436268 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f888995-db08-4fec-a7b6-b1f2fbfcdb2f None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:53.478741 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:53.509974 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2479afb3-2de0-488a-a0db-35c8c5db6a08 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:53.539037 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:53.543187 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f888995-db08-4fec-a7b6-b1f2fbfcdb2f None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:53.546258 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b67c1b-2ebc-4d0b-8a62-a6dea55dbd72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:53.546868 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b67c1b-2ebc-4d0b-8a62-a6dea55dbd72 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:53.547142 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b67c1b-2ebc-4d0b-8a62-a6dea55dbd72 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:53.547411 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b67c1b-2ebc-4d0b-8a62-a6dea55dbd72 None admin] PATH_INFO: `/v3/services` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:53.548028 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0b67c1b-2ebc-4d0b-8a62-a6dea55dbd72 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:53.548330 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0b67c1b-2ebc-4d0b-8a62-a6dea55dbd72 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:53.549869 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0b67c1b-2ebc-4d0b-8a62-a6dea55dbd72 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:53.556051 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 456/896] 10.222.0.52 () {56 vars in 1126 bytes} [Sun Sep 25 00:42:53 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:53.556994 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b67c1b-2ebc-4d0b-8a62-a6dea55dbd72 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:53.588358 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d92e72-bb31-4c84-8a87-6f5d010007ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:53.589222 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d92e72-bb31-4c84-8a87-6f5d010007ac None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:53.589559 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d92e72-bb31-4c84-8a87-6f5d010007ac None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:53.589904 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d92e72-bb31-4c84-8a87-6f5d010007ac None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:53.598957 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26d92e72-bb31-4c84-8a87-6f5d010007ac None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:53.600981 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26d92e72-bb31-4c84-8a87-6f5d010007ac None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:53.614361 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-26d92e72-bb31-4c84-8a87-6f5d010007ac None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 25 00:42:53.614361 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:53.614361 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:53.614361 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:53.614361 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:53.614361 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:53.614361 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:53.614361 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:53.614361 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:53.614361 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:53.614361 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:53.614361 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:53.614361 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 00:42:53.614361 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 00:42:53.614361 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 00:42:53.614361 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 00:42:53.615982 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:53.615982 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:53.615982 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 00:42:53.615982 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:53.615982 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 00:42:53.615982 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:53.615982 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:53.615982 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:53.615982 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:53.615982 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:53.615982 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:53.615982 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:53.615982 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:53.615982 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:53.615982 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:53.615982 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:53.615982 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:53.615982 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:53.617454 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:53.617454 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:53.617454 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 00:42:53.617454 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 00:42:53.617454 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 00:42:53.617454 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 00:42:53.617454 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 00:42:53.617454 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 00:42:53.617454 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 25 00:42:53.617454 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:53.617454 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 441/897] 10.222.0.52 () {56 vars in 1117 bytes} [Sun Sep 25 00:42:53 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:53.617454 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d92e72-bb31-4c84-8a87-6f5d010007ac None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:53.666384 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b67c1b-2ebc-4d0b-8a62-a6dea55dbd72 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:53.702339 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-777bf8d3-aa28-4947-bac0-3eb0787f92cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:53.702955 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777bf8d3-aa28-4947-bac0-3eb0787f92cb None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:53.703229 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777bf8d3-aa28-4947-bac0-3eb0787f92cb None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:53.703502 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777bf8d3-aa28-4947-bac0-3eb0787f92cb None admin] PATH_INFO: `/v3/endpoints` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:53.704199 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-777bf8d3-aa28-4947-bac0-3eb0787f92cb None admin] RBAC: Adding query filter params (interface=public, service_id=a266571533fc4119be3454834b16984e, region_id=RegionOne) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:53.704505 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-777bf8d3-aa28-4947-bac0-3eb0787f92cb None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:53.705898 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-777bf8d3-aa28-4947-bac0-3eb0787f92cb None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:53.711789 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 457/898] 10.222.0.52 () {56 vars in 1251 bytes} [Sun Sep 25 00:42:53 2022] GET /identity/v3/endpoints?service_id=a266571533fc4119be3454834b16984e&interface=public®ion_id=RegionOne => generated 194 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:53.712473 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-777bf8d3-aa28-4947-bac0-3eb0787f92cb None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:53.715242 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:53.725863 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d92e72-bb31-4c84-8a87-6f5d010007ac None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:53.760520 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9183ebf5-d225-4998-8526-ee291c5d3e60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:53.761121 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9183ebf5-d225-4998-8526-ee291c5d3e60 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:53.761432 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9183ebf5-d225-4998-8526-ee291c5d3e60 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:53.761695 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9183ebf5-d225-4998-8526-ee291c5d3e60 None admin] PATH_INFO: `/v3/services` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:53.762314 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9183ebf5-d225-4998-8526-ee291c5d3e60 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:53.762608 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9183ebf5-d225-4998-8526-ee291c5d3e60 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:53.763977 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9183ebf5-d225-4998-8526-ee291c5d3e60 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:53.768453 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 442/899] 10.222.0.52 () {56 vars in 1122 bytes} [Sun Sep 25 00:42:53 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:53.769225 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9183ebf5-d225-4998-8526-ee291c5d3e60 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:53.799082 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:53.842700 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-777bf8d3-aa28-4947-bac0-3eb0787f92cb None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:53.883912 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f466e5-dc5a-48ec-af67-c90fb3b527b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:53.884533 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f466e5-dc5a-48ec-af67-c90fb3b527b7 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:53.884811 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f466e5-dc5a-48ec-af67-c90fb3b527b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:53.885076 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f466e5-dc5a-48ec-af67-c90fb3b527b7 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:53.885816 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67f466e5-dc5a-48ec-af67-c90fb3b527b7 None admin] RBAC: Adding query filter params (interface=public, service_id=5d59b98b779142858698d14efce69c29, region_id=RegionOne) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:53.886120 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67f466e5-dc5a-48ec-af67-c90fb3b527b7 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:53.887508 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67f466e5-dc5a-48ec-af67-c90fb3b527b7 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:53.893291 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 458/900] 10.222.0.52 () {56 vars in 1251 bytes} [Sun Sep 25 00:42:53 2022] GET /identity/v3/endpoints?service_id=5d59b98b779142858698d14efce69c29&interface=public®ion_id=RegionOne => generated 194 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:53.898773 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f466e5-dc5a-48ec-af67-c90fb3b527b7 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:53.903115 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9183ebf5-d225-4998-8526-ee291c5d3e60 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:53.947480 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abefce55-8345-40b8-b91f-e27eef2ff9fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:53.948217 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abefce55-8345-40b8-b91f-e27eef2ff9fd None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:53.948496 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abefce55-8345-40b8-b91f-e27eef2ff9fd None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:53.948768 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abefce55-8345-40b8-b91f-e27eef2ff9fd None admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:53.949628 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abefce55-8345-40b8-b91f-e27eef2ff9fd None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:53.949938 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abefce55-8345-40b8-b91f-e27eef2ff9fd None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:53.951434 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abefce55-8345-40b8-b91f-e27eef2ff9fd None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:53.981812 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 443/901] 10.222.0.52 () {56 vars in 1149 bytes} [Sun Sep 25 00:42:53 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:53.982672 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abefce55-8345-40b8-b91f-e27eef2ff9fd None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:54.004993 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:54.036171 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f466e5-dc5a-48ec-af67-c90fb3b527b7 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:54.080573 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be217f7f-6cb0-4a64-a961-a54548bc8ea3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:54.081230 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be217f7f-6cb0-4a64-a961-a54548bc8ea3 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:54.081543 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be217f7f-6cb0-4a64-a961-a54548bc8ea3 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:54.081806 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be217f7f-6cb0-4a64-a961-a54548bc8ea3 None admin] PATH_INFO: `/v3/services` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:54.082418 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be217f7f-6cb0-4a64-a961-a54548bc8ea3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:54.083808 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be217f7f-6cb0-4a64-a961-a54548bc8ea3 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:54.089104 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:54.091980 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 459/902] 10.222.0.52 () {56 vars in 1087 bytes} [Sun Sep 25 00:42:53 2022] GET /identity/v3/services => generated 1268 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:42:54.093124 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be217f7f-6cb0-4a64-a961-a54548bc8ea3 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:54.123432 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abefce55-8345-40b8-b91f-e27eef2ff9fd None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:54.183931 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-215ad006-acc5-4aa1-8b82-6e848f4f2964 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:54.185439 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215ad006-acc5-4aa1-8b82-6e848f4f2964 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:54.185439 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215ad006-acc5-4aa1-8b82-6e848f4f2964 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:54.185439 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215ad006-acc5-4aa1-8b82-6e848f4f2964 None admin] PATH_INFO: `/v3/services` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:54.186052 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-215ad006-acc5-4aa1-8b82-6e848f4f2964 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:54.188061 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-215ad006-acc5-4aa1-8b82-6e848f4f2964 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:54.196773 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 444/903] 10.222.0.52 () {56 vars in 1087 bytes} [Sun Sep 25 00:42:53 2022] GET /identity/v3/services => generated 1268 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:42:54.232865 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be217f7f-6cb0-4a64-a961-a54548bc8ea3 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:54.273170 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:54.281455 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa04a222-5058-4fa4-bd27-006ff545d0e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:54.282446 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa04a222-5058-4fa4-bd27-006ff545d0e6 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:54.282898 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa04a222-5058-4fa4-bd27-006ff545d0e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:54.283313 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa04a222-5058-4fa4-bd27-006ff545d0e6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:54.375446 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa04a222-5058-4fa4-bd27-006ff545d0e6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:54.376819 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa04a222-5058-4fa4-bd27-006ff545d0e6 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:54.493310 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 460/904] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:42:54 2022] GET /identity/v3/auth/tokens => generated 1372 bytes in 401 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 25 00:42:54.502977 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-215ad006-acc5-4aa1-8b82-6e848f4f2964 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:54.622262 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-215ad006-acc5-4aa1-8b82-6e848f4f2964 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:54.661238 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d6b1554-06d5-4290-adbd-5ebc31d21f97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:54.661818 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d6b1554-06d5-4290-adbd-5ebc31d21f97 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:54.662623 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d6b1554-06d5-4290-adbd-5ebc31d21f97 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:54.663043 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d6b1554-06d5-4290-adbd-5ebc31d21f97 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:54.674589 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d6b1554-06d5-4290-adbd-5ebc31d21f97 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:54.676037 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d6b1554-06d5-4290-adbd-5ebc31d21f97 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:54.679393 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:54.692377 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-6d6b1554-06d5-4290-adbd-5ebc31d21f97 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:54.692377 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:54.692377 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:54.692377 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:54.692377 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:54.692377 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:54.692377 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:54.692377 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:54.692377 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:54.692377 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:54.692377 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:54.692377 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:54.692377 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:54.692377 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:54.692377 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:54.692377 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:54.694002 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:54.694002 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:54.694002 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:54.694002 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:54.694002 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:54.694002 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:54.694002 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:54.694002 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:54.694002 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:54.694002 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:54.694002 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:54.694002 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:54.694002 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:54.694002 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:54.694002 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:54.694002 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:54.694002 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:54.694929 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:54.694929 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:54.694929 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:54.694929 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 445/905] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:54.702388 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa04a222-5058-4fa4-bd27-006ff545d0e6 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:54.735486 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:54.815616 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa04a222-5058-4fa4-bd27-006ff545d0e6 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:54.857789 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1911d7-dcc0-4653-8f85-b62c247dc1c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:54.858767 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1911d7-dcc0-4653-8f85-b62c247dc1c4 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:54.859271 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1911d7-dcc0-4653-8f85-b62c247dc1c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:54.859766 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1911d7-dcc0-4653-8f85-b62c247dc1c4 None admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:54.860608 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f1911d7-dcc0-4653-8f85-b62c247dc1c4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:54.861120 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f1911d7-dcc0-4653-8f85-b62c247dc1c4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:54.862713 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f1911d7-dcc0-4653-8f85-b62c247dc1c4 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:54.876739 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 461/906] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:54.882793 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d6b1554-06d5-4290-adbd-5ebc31d21f97 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:54.994509 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d6b1554-06d5-4290-adbd-5ebc31d21f97 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:55.036494 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66384a0f-606b-4aef-95e2-8a77cee75279 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:55.037196 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66384a0f-606b-4aef-95e2-8a77cee75279 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:55.037496 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66384a0f-606b-4aef-95e2-8a77cee75279 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:55.037760 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66384a0f-606b-4aef-95e2-8a77cee75279 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:55.045409 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66384a0f-606b-4aef-95e2-8a77cee75279 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:55.046874 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66384a0f-606b-4aef-95e2-8a77cee75279 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:55.059766 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-66384a0f-606b-4aef-95e2-8a77cee75279 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:42:55.059766 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:55.059766 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:55.059766 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:55.059766 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:55.059766 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:55.059766 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:55.059766 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:55.059766 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:55.059766 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:55.059766 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:55.059766 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:55.059766 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 00:42:55.059766 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 00:42:55.059766 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 00:42:55.060744 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 00:42:55.060744 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:55.060744 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:55.060744 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:55.060744 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:55.060744 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:55.060744 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:55.060744 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:55.060744 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:55.060744 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:55.060744 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:55.060744 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:55.060744 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:55.060744 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:55.060744 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:55.060744 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:55.060744 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:55.061670 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 00:42:55.061670 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 00:42:55.061670 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 00:42:55.061670 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 00:42:55.061670 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 00:42:55.061670 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 00:42:55.061670 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:42:55.061670 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:55.061670 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 446/907] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:42:54 2022] GET /identity/v3/projects/service => generated 88 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:55.068611 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1911d7-dcc0-4653-8f85-b62c247dc1c4 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:55.178888 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:55.189763 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1911d7-dcc0-4653-8f85-b62c247dc1c4 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:55.228413 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8fbb9df-c6e0-42cb-bf7c-c1b95c9db7bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:55.229485 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8fbb9df-c6e0-42cb-bf7c-c1b95c9db7bd None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:55.230155 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8fbb9df-c6e0-42cb-bf7c-c1b95c9db7bd None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:55.230647 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8fbb9df-c6e0-42cb-bf7c-c1b95c9db7bd None admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:55.231580 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8fbb9df-c6e0-42cb-bf7c-c1b95c9db7bd None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:55.232153 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8fbb9df-c6e0-42cb-bf7c-c1b95c9db7bd None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:55.234000 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8fbb9df-c6e0-42cb-bf7c-c1b95c9db7bd None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:55.248876 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 462/908] 10.222.0.52 () {56 vars in 1148 bytes} [Sun Sep 25 00:42:55 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:55.255590 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66384a0f-606b-4aef-95e2-8a77cee75279 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:55.319296 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:55.369231 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66384a0f-606b-4aef-95e2-8a77cee75279 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:55.405931 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe4edd9-876a-45cf-ae62-c2794366fd75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:55.406562 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe4edd9-876a-45cf-ae62-c2794366fd75 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:55.406831 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe4edd9-876a-45cf-ae62-c2794366fd75 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:55.407113 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe4edd9-876a-45cf-ae62-c2794366fd75 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:55.407825 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fe4edd9-876a-45cf-ae62-c2794366fd75 None admin] RBAC: Adding query filter params (role.id=5d162233989d4e439c441608704dc9b0, scope.project.id=9db0fd07fc494203b2ada13f3cd046d3, user.id=3f20ffd8df994c1db1b1e7248db5b5a3) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:55.408123 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fe4edd9-876a-45cf-ae62-c2794366fd75 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:55.409574 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fe4edd9-876a-45cf-ae62-c2794366fd75 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:55.416888 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 447/909] 10.222.0.52 () {56 vars in 1374 bytes} [Sun Sep 25 00:42:55 2022] GET /identity/v3/role_assignments?user.id=3f20ffd8df994c1db1b1e7248db5b5a3&scope.project.id=9db0fd07fc494203b2ada13f3cd046d3&role.id=5d162233989d4e439c441608704dc9b0 => generated 259 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:56.225293 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4ad6fa-d477-499d-8e59-e064d4970002 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:56.227935 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4ad6fa-d477-499d-8e59-e064d4970002 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:56.227935 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4ad6fa-d477-499d-8e59-e064d4970002 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:56.227935 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 463/910] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:42:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:42:56.234207 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae389fa8-87f6-4ad4-af25-dc2b441190db None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:56.234717 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae389fa8-87f6-4ad4-af25-dc2b441190db None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:56.234717 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae389fa8-87f6-4ad4-af25-dc2b441190db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:56.269706 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-ae389fa8-87f6-4ad4-af25-dc2b441190db None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:42:56.270938 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-ae389fa8-87f6-4ad4-af25-dc2b441190db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:42:56.415849 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 448/911] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:42:56 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:42:56.610153 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a39d535-be8f-4365-99a0-59ef4a99943f None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:56.610619 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a39d535-be8f-4365-99a0-59ef4a99943f None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:56.611102 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a39d535-be8f-4365-99a0-59ef4a99943f None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:56.614331 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 464/912] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:42:56 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:42:56.623904 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399b356e-7d70-4ecc-ae60-033dff0de044 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:56.624382 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399b356e-7d70-4ecc-ae60-033dff0de044 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:56.624865 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399b356e-7d70-4ecc-ae60-033dff0de044 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:56.654594 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-399b356e-7d70-4ecc-ae60-033dff0de044 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:42:56.656474 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-399b356e-7d70-4ecc-ae60-033dff0de044 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:42:56.712435 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83024a64-2cc4-4575-b7d2-740040008736 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:56.712766 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83024a64-2cc4-4575-b7d2-740040008736 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:56.712965 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83024a64-2cc4-4575-b7d2-740040008736 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:56.714570 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 465/913] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:42:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 00:42:56.719638 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10424c3-5b9e-4c2c-a20c-8d4f1995f86c None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:56.719915 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10424c3-5b9e-4c2c-a20c-8d4f1995f86c None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:56.720182 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10424c3-5b9e-4c2c-a20c-8d4f1995f86c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:56.730869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:56.742727 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-f10424c3-5b9e-4c2c-a20c-8d4f1995f86c None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:42:56.743830 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-f10424c3-5b9e-4c2c-a20c-8d4f1995f86c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:42:56.754074 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 449/914] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:42:56 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:42:56.756935 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-399b356e-7d70-4ecc-ae60-033dff0de044 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:56.849100 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 466/915] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:42:56 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 25 00:42:56.894458 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-399b356e-7d70-4ecc-ae60-033dff0de044 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:56.931316 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf47ca7-9ac7-456d-868a-786f99a6313f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:56.932242 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf47ca7-9ac7-456d-868a-786f99a6313f None admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:56.932663 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf47ca7-9ac7-456d-868a-786f99a6313f None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:56.933069 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf47ca7-9ac7-456d-868a-786f99a6313f None admin] PATH_INFO: `/v3/services` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:56.933891 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bf47ca7-9ac7-456d-868a-786f99a6313f None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:56.935457 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bf47ca7-9ac7-456d-868a-786f99a6313f None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:56.953734 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 450/916] 10.222.0.52 () {60 vars in 1141 bytes} [Sun Sep 25 00:42:56 2022] POST /identity/v3/services => generated 241 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:42:57.029142 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f10424c3-5b9e-4c2c-a20c-8d4f1995f86c None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:57.133070 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf47ca7-9ac7-456d-868a-786f99a6313f None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:57.134618 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:57.162542 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f10424c3-5b9e-4c2c-a20c-8d4f1995f86c None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:57.193664 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:57.232275 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31916a81-fd76-4de7-9a13-54ba74c040c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:57.233371 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31916a81-fd76-4de7-9a13-54ba74c040c9 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:57.234557 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31916a81-fd76-4de7-9a13-54ba74c040c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:57.235032 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31916a81-fd76-4de7-9a13-54ba74c040c9 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:57.236031 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31916a81-fd76-4de7-9a13-54ba74c040c9 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:57.239016 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31916a81-fd76-4de7-9a13-54ba74c040c9 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:57.258834 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-31916a81-fd76-4de7-9a13-54ba74c040c9 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 25 00:42:57.258834 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:57.258834 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 00:42:57.258834 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 00:42:57.258834 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 00:42:57.258834 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 00:42:57.258834 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 00:42:57.258834 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:42:57.258834 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 25 00:42:57.258834 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:57.258834 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 00:42:57.258834 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:57.258834 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:57.258834 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:57.258834 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:57.258834 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:57.258834 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:57.260776 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:57.260776 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:57.260776 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:57.260776 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:57.260776 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:57.260776 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:57.260776 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 00:42:57.260776 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 00:42:57.260776 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 00:42:57.260776 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 00:42:57.260776 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:57.260776 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:57.260776 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:57.260776 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:57.260776 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:57.260776 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:57.262449 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:57.262449 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:57.262449 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:57.262449 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:57.262449 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:57.262449 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:57.262449 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:57.262449 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:57.262449 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:57.262449 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:57.262449 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 00:42:57.262449 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:42:57.262449 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 25 00:42:57.262449 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:42:57.266588 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 467/917] 10.222.0.52 () {56 vars in 1114 bytes} [Sun Sep 25 00:42:57 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 233 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:42:57.272546 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31916a81-fd76-4de7-9a13-54ba74c040c9 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:57.306430 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf47ca7-9ac7-456d-868a-786f99a6313f None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:57.367038 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641e0749-9dd9-4021-b650-6fa0f4cd45c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:57.368527 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641e0749-9dd9-4021-b650-6fa0f4cd45c8 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:57.369333 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641e0749-9dd9-4021-b650-6fa0f4cd45c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:57.370081 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641e0749-9dd9-4021-b650-6fa0f4cd45c8 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:57.371330 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-641e0749-9dd9-4021-b650-6fa0f4cd45c8 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:57.374137 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-641e0749-9dd9-4021-b650-6fa0f4cd45c8 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:57.391258 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-641e0749-9dd9-4021-b650-6fa0f4cd45c8 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 25 00:42:57.391258 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:57.391258 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 00:42:57.391258 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 00:42:57.391258 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 00:42:57.391258 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 00:42:57.391258 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 00:42:57.391258 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:42:57.391258 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 25 00:42:57.391258 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:57.391258 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 00:42:57.391258 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:57.391258 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:57.391258 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:57.391258 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:57.391258 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:57.393162 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:57.393162 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:57.393162 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:57.393162 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:57.393162 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:57.393162 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:57.393162 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:57.393162 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 00:42:57.393162 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 00:42:57.393162 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 00:42:57.393162 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 00:42:57.393162 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:57.393162 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:57.393162 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:57.393162 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:57.393162 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:57.393162 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:57.395062 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:57.395062 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:57.395062 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:57.395062 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:57.395062 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:57.395062 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:57.395062 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:57.395062 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:57.395062 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:57.395062 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:57.395062 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 00:42:57.395062 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:42:57.395062 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 25 00:42:57.395062 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:57.408651 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 451/918] 10.222.0.52 () {56 vars in 1132 bytes} [Sun Sep 25 00:42:57 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 268 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:42:57.410655 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641e0749-9dd9-4021-b650-6fa0f4cd45c8 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:57.467850 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31916a81-fd76-4de7-9a13-54ba74c040c9 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:57.524480 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c4c8af-50f4-42b7-8998-5d03fd89a9ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:57.525097 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c4c8af-50f4-42b7-8998-5d03fd89a9ac None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:57.525404 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c4c8af-50f4-42b7-8998-5d03fd89a9ac None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:57.525671 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c4c8af-50f4-42b7-8998-5d03fd89a9ac None admin] PATH_INFO: `/v3/services` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:57.526588 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3c4c8af-50f4-42b7-8998-5d03fd89a9ac None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:57.526889 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3c4c8af-50f4-42b7-8998-5d03fd89a9ac None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:57.528529 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3c4c8af-50f4-42b7-8998-5d03fd89a9ac None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:57.537485 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 468/919] 10.222.0.52 () {56 vars in 1114 bytes} [Sun Sep 25 00:42:57 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:42:57.543232 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c4c8af-50f4-42b7-8998-5d03fd89a9ac None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:57.548596 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:57.558213 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641e0749-9dd9-4021-b650-6fa0f4cd45c8 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:57.562276 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:57.610386 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29621a00-408c-44f6-998f-f8794565460b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:57.611039 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29621a00-408c-44f6-998f-f8794565460b None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:57.611296 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29621a00-408c-44f6-998f-f8794565460b None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:57.611563 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29621a00-408c-44f6-998f-f8794565460b None admin] PATH_INFO: `/v3/services` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:57.612187 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29621a00-408c-44f6-998f-f8794565460b None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:57.612479 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29621a00-408c-44f6-998f-f8794565460b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:57.614788 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29621a00-408c-44f6-998f-f8794565460b None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:57.627407 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 452/920] 10.222.0.52 () {56 vars in 1126 bytes} [Sun Sep 25 00:42:57 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:57.628479 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29621a00-408c-44f6-998f-f8794565460b None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:57.681944 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c4c8af-50f4-42b7-8998-5d03fd89a9ac None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:57.735472 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-445a2616-303f-4ccc-9fe7-0cadd0c556a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:57.736088 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445a2616-303f-4ccc-9fe7-0cadd0c556a6 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:57.736362 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445a2616-303f-4ccc-9fe7-0cadd0c556a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:57.736629 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445a2616-303f-4ccc-9fe7-0cadd0c556a6 None admin] PATH_INFO: `/v3/services` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:57.737278 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-445a2616-303f-4ccc-9fe7-0cadd0c556a6 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:57.737583 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-445a2616-303f-4ccc-9fe7-0cadd0c556a6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:57.739216 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-445a2616-303f-4ccc-9fe7-0cadd0c556a6 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:57.749474 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 469/921] 10.222.0.52 () {56 vars in 1114 bytes} [Sun Sep 25 00:42:57 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:57.753796 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-445a2616-303f-4ccc-9fe7-0cadd0c556a6 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:57.762631 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29621a00-408c-44f6-998f-f8794565460b None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:57.810446 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4e0b965-6be0-48d6-b4f4-b6deaa9f66d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:57.810969 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e0b965-6be0-48d6-b4f4-b6deaa9f66d9 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:57.811236 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e0b965-6be0-48d6-b4f4-b6deaa9f66d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:57.811498 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e0b965-6be0-48d6-b4f4-b6deaa9f66d9 None admin] PATH_INFO: `/v3/services` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:57.812119 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4e0b965-6be0-48d6-b4f4-b6deaa9f66d9 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:57.812413 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4e0b965-6be0-48d6-b4f4-b6deaa9f66d9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:57.813819 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4e0b965-6be0-48d6-b4f4-b6deaa9f66d9 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:57.819846 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 453/922] 10.222.0.52 () {56 vars in 1126 bytes} [Sun Sep 25 00:42:57 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:57.824872 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb7493f-936c-47b3-aca0-2743417d1deb None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:57.824872 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb7493f-936c-47b3-aca0-2743417d1deb None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:57.824872 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb7493f-936c-47b3-aca0-2743417d1deb None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:57.824872 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 454/923] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:42:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:42:57.825820 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb7493f-936c-47b3-aca0-2743417d1deb None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:57.838754 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:57.862192 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:57.883344 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-445a2616-303f-4ccc-9fe7-0cadd0c556a6 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:57.921421 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a57eaf7-b19d-460a-8e84-802fb6e4e37b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:57.922044 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a57eaf7-b19d-460a-8e84-802fb6e4e37b None admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:57.922324 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a57eaf7-b19d-460a-8e84-802fb6e4e37b None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:57.922593 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a57eaf7-b19d-460a-8e84-802fb6e4e37b None admin] PATH_INFO: `/v3/endpoints` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:57.923293 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a57eaf7-b19d-460a-8e84-802fb6e4e37b None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:57.924701 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a57eaf7-b19d-460a-8e84-802fb6e4e37b None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:57.957295 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 470/924] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:42:57 2022] POST /identity/v3/endpoints => generated 346 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:42:57.957295 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2440189d-e6af-43e4-b702-126f1567f17a None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:57.957295 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2440189d-e6af-43e4-b702-126f1567f17a None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:57.957295 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2440189d-e6af-43e4-b702-126f1567f17a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:57.967432 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb7493f-936c-47b3-aca0-2743417d1deb None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:57.976560 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-2440189d-e6af-43e4-b702-126f1567f17a None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:42:57.977850 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-2440189d-e6af-43e4-b702-126f1567f17a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:42:58.017049 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0125c9-64ae-438f-ae3d-c6639f8ec6c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:58.018212 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0125c9-64ae-438f-ae3d-c6639f8ec6c7 None admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:58.018576 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0125c9-64ae-438f-ae3d-c6639f8ec6c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:58.018894 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0125c9-64ae-438f-ae3d-c6639f8ec6c7 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:58.019762 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a0125c9-64ae-438f-ae3d-c6639f8ec6c7 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:58.021976 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a0125c9-64ae-438f-ae3d-c6639f8ec6c7 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:58.061531 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 455/925] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:42:57 2022] POST /identity/v3/endpoints => generated 347 bytes in 236 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:42:58.081043 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 471/926] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:42:57 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:42:58.365369 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0125c9-64ae-438f-ae3d-c6639f8ec6c7 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:58.459661 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:58.479000 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0125c9-64ae-438f-ae3d-c6639f8ec6c7 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:58.516636 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d7e8d1-b832-4ef7-944a-82fb26fd2123 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:58.517289 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d7e8d1-b832-4ef7-944a-82fb26fd2123 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:58.517582 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d7e8d1-b832-4ef7-944a-82fb26fd2123 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:58.517852 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d7e8d1-b832-4ef7-944a-82fb26fd2123 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:58.529075 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81d7e8d1-b832-4ef7-944a-82fb26fd2123 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:42:58.534437 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-81d7e8d1-b832-4ef7-944a-82fb26fd2123 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 25 00:42:58.534437 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:58.534437 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:58.534437 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:58.534437 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:58.534437 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:58.534437 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:58.534437 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:58.534437 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:58.534437 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:58.534437 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:58.534437 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:58.534437 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:42:58.534437 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:42:58.534437 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:42:58.534437 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:42:58.535379 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:42:58.535379 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:42:58.535379 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:42:58.535379 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:42:58.535379 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:58.535379 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:58.535379 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:58.535379 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:58.535379 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:58.535379 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:58.535379 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:58.535379 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:58.535379 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:58.535379 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:58.535379 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:58.535379 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:58.536272 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:58.536272 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:58.536272 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:58.536272 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:58.536272 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:42:58.536272 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:42:58.536272 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:42:58.536272 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:42:58.536272 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:42:58.536272 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:42:58.536272 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 25 00:42:58.536272 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:58.536272 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 456/927] 10.222.0.52 () {56 vars in 1120 bytes} [Sun Sep 25 00:42:58 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:58.542110 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2440189d-e6af-43e4-b702-126f1567f17a None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:58.597773 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:58.654561 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2440189d-e6af-43e4-b702-126f1567f17a None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:58.695110 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d557829-9f78-4548-a21f-1d93f02b2b74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:58.695747 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d557829-9f78-4548-a21f-1d93f02b2b74 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:58.696021 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d557829-9f78-4548-a21f-1d93f02b2b74 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:58.696285 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d557829-9f78-4548-a21f-1d93f02b2b74 None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:58.696952 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d557829-9f78-4548-a21f-1d93f02b2b74 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:58.697251 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d557829-9f78-4548-a21f-1d93f02b2b74 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:58.698644 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d557829-9f78-4548-a21f-1d93f02b2b74 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:58.708097 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 472/928] 10.222.0.52 () {56 vars in 1115 bytes} [Sun Sep 25 00:42:58 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:58.714071 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d7e8d1-b832-4ef7-944a-82fb26fd2123 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:58.830359 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d7e8d1-b832-4ef7-944a-82fb26fd2123 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:58.871806 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9dbb473-30b3-4afd-93a7-c11c563eaa99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:58.872467 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9dbb473-30b3-4afd-93a7-c11c563eaa99 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:58.872991 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9dbb473-30b3-4afd-93a7-c11c563eaa99 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:58.873285 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9dbb473-30b3-4afd-93a7-c11c563eaa99 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:58.886708 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9dbb473-30b3-4afd-93a7-c11c563eaa99 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:58.888246 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9dbb473-30b3-4afd-93a7-c11c563eaa99 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:58.891491 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:58.905676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-d9dbb473-30b3-4afd-93a7-c11c563eaa99 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:58.905676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:58.905676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:58.905676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:58.905676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:58.905676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:58.905676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:58.905676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:58.905676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:58.905676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:58.905676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:58.905676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:58.905676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:58.905676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:58.905676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:58.905676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:58.906895 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:58.906895 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:58.906895 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:58.906895 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:58.906895 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:58.906895 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:58.906895 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:58.906895 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:58.906895 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:58.906895 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:58.906895 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:58.906895 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:58.906895 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:58.906895 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:58.906895 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:58.906895 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:58.906895 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:58.907819 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:58.907819 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:58.907819 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:58.910167 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 457/929] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:58.916084 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d557829-9f78-4548-a21f-1d93f02b2b74 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:59.029636 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d557829-9f78-4548-a21f-1d93f02b2b74 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:59.045808 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:59.071492 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfdea854-3a1a-44ec-affd-4c58482c84d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:59.072164 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfdea854-3a1a-44ec-affd-4c58482c84d2 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:59.072443 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfdea854-3a1a-44ec-affd-4c58482c84d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:59.072709 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfdea854-3a1a-44ec-affd-4c58482c84d2 None admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:59.073391 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfdea854-3a1a-44ec-affd-4c58482c84d2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:59.073697 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfdea854-3a1a-44ec-affd-4c58482c84d2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:59.075081 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfdea854-3a1a-44ec-affd-4c58482c84d2 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:59.088532 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 473/930] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:59.094273 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9dbb473-30b3-4afd-93a7-c11c563eaa99 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:59.207475 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9dbb473-30b3-4afd-93a7-c11c563eaa99 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:59.247521 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e5e290d-a893-4aa6-951b-d2e20fb99735 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:59.248214 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e5e290d-a893-4aa6-951b-d2e20fb99735 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:59.248765 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e5e290d-a893-4aa6-951b-d2e20fb99735 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:59.249034 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e5e290d-a893-4aa6-951b-d2e20fb99735 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:59.257832 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e5e290d-a893-4aa6-951b-d2e20fb99735 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:59.259278 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e5e290d-a893-4aa6-951b-d2e20fb99735 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:59.271220 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-9e5e290d-a893-4aa6-951b-d2e20fb99735 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 25 00:42:59.271220 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:59.271220 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:59.271220 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:59.271220 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:59.271220 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:59.271220 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:59.271220 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:59.271220 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:59.271220 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:59.271220 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:59.271220 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:59.271220 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 00:42:59.271220 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 00:42:59.271220 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 00:42:59.271220 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 00:42:59.272625 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:59.272625 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:59.272625 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 00:42:59.272625 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:59.272625 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 00:42:59.272625 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:42:59.272625 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:59.272625 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:59.272625 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:59.272625 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:59.272625 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:59.272625 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:59.272625 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:59.272625 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:59.272625 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:59.272625 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:59.272625 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:59.272625 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:59.274246 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:59.274246 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:59.274246 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 00:42:59.274246 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 00:42:59.274246 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 00:42:59.274246 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 00:42:59.274246 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 00:42:59.274246 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 00:42:59.274246 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 25 00:42:59.274246 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:59.274246 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 458/931] 10.222.0.52 () {56 vars in 1117 bytes} [Sun Sep 25 00:42:59 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:59.278663 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfdea854-3a1a-44ec-affd-4c58482c84d2 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:59.336509 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b7b275-b704-40ff-a9a7-03a4e18695d4 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:59.336829 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b7b275-b704-40ff-a9a7-03a4e18695d4 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:59.337056 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b7b275-b704-40ff-a9a7-03a4e18695d4 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:59.338689 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 459/932] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:42:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 00:42:59.343987 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4811fe53-a364-4bb7-b7f9-ec59b9100899 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:59.344265 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4811fe53-a364-4bb7-b7f9-ec59b9100899 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:59.344519 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4811fe53-a364-4bb7-b7f9-ec59b9100899 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:59.365597 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-4811fe53-a364-4bb7-b7f9-ec59b9100899 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:42:59.367173 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-4811fe53-a364-4bb7-b7f9-ec59b9100899 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:42:59.399720 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfdea854-3a1a-44ec-affd-4c58482c84d2 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:59.426437 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:59.450943 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b7fc16-776f-407e-af3d-562b510cf6e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:59.451910 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b7fc16-776f-407e-af3d-562b510cf6e3 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:59.452391 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b7fc16-776f-407e-af3d-562b510cf6e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:59.452888 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b7fc16-776f-407e-af3d-562b510cf6e3 None admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:59.453602 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 460/933] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:42:59 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 25 00:42:59.454109 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05b7fc16-776f-407e-af3d-562b510cf6e3 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:59.454614 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05b7fc16-776f-407e-af3d-562b510cf6e3 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:59.456258 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05b7fc16-776f-407e-af3d-562b510cf6e3 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:59.477520 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 474/934] 10.222.0.52 () {56 vars in 1149 bytes} [Sun Sep 25 00:42:59 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:59.488945 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4811fe53-a364-4bb7-b7f9-ec59b9100899 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:59.606256 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4811fe53-a364-4bb7-b7f9-ec59b9100899 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:59.643662 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edc3af7d-38d2-44f6-8447-70963021c35e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:59.644687 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc3af7d-38d2-44f6-8447-70963021c35e None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:59.645171 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc3af7d-38d2-44f6-8447-70963021c35e None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:59.645698 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc3af7d-38d2-44f6-8447-70963021c35e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:59.657370 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edc3af7d-38d2-44f6-8447-70963021c35e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:59.659130 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edc3af7d-38d2-44f6-8447-70963021c35e None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:59.674911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-edc3af7d-38d2-44f6-8447-70963021c35e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:59.674911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:59.674911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:59.674911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:59.674911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:59.674911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:59.674911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:59.674911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:59.674911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:59.674911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:59.674911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:59.674911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:59.674911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:42:59.674911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:42:59.674911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:42:59.674911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:42:59.675924 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:59.675924 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:59.675924 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:59.675924 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:59.675924 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:59.675924 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:59.675924 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:59.675924 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:59.675924 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:59.675924 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:59.675924 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:59.675924 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:59.675924 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:59.675924 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:59.675924 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:59.675924 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:59.675924 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:42:59.676916 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:42:59.676916 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:42:59.676916 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:59.679341 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 461/935] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:42:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:59.685240 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b7fc16-776f-407e-af3d-562b510cf6e3 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:59.720745 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:59.744162 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edc3af7d-38d2-44f6-8447-70963021c35e None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:59.792215 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:42:59.818726 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b7fc16-776f-407e-af3d-562b510cf6e3 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:59.871772 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edc3af7d-38d2-44f6-8447-70963021c35e None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:42:59.872099 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1d48a3-1506-4060-8198-7320d5257597 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:59.872642 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1d48a3-1506-4060-8198-7320d5257597 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:59.872921 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1d48a3-1506-4060-8198-7320d5257597 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:59.873188 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1d48a3-1506-4060-8198-7320d5257597 None admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:59.873864 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1d48a3-1506-4060-8198-7320d5257597 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:42:59.874165 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1d48a3-1506-4060-8198-7320d5257597 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:59.876895 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1d48a3-1506-4060-8198-7320d5257597 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:59.893509 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 475/936] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:42:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:42:59.899799 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1d48a3-1506-4060-8198-7320d5257597 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:42:59.918696 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-319d144c-d643-44cc-80e8-847b7342bb31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:42:59.919344 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-319d144c-d643-44cc-80e8-847b7342bb31 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:42:59.919620 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-319d144c-d643-44cc-80e8-847b7342bb31 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:42:59.919889 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-319d144c-d643-44cc-80e8-847b7342bb31 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:42:59.920535 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-319d144c-d643-44cc-80e8-847b7342bb31 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:42:59.922062 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-319d144c-d643-44cc-80e8-847b7342bb31 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:42:59.932817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-319d144c-d643-44cc-80e8-847b7342bb31 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 25 00:42:59.932817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:59.932817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 00:42:59.932817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 00:42:59.932817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 00:42:59.932817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 00:42:59.932817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 00:42:59.932817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:42:59.932817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 25 00:42:59.932817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:59.932817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 00:42:59.932817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:59.932817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:42:59.932817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:42:59.932817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:42:59.932817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:42:59.932817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:42:59.933937 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:42:59.933937 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:42:59.933937 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:42:59.933937 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:42:59.933937 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:42:59.933937 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:42:59.933937 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 00:42:59.933937 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 00:42:59.933937 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 00:42:59.933937 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 00:42:59.933937 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:42:59.933937 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:42:59.933937 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:42:59.933937 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:42:59.933937 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:42:59.933937 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:42:59.934796 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:42:59.934796 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:42:59.934796 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:42:59.934796 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:42:59.934796 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:42:59.934796 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:42:59.934796 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:42:59.934796 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:42:59.934796 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:42:59.934796 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:42:59.934796 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 00:42:59.934796 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:42:59.934796 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 25 00:42:59.934796 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:42:59.939298 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 462/937] 10.222.0.52 () {56 vars in 1126 bytes} [Sun Sep 25 00:42:59 2022] GET /identity/v3/services/object-store => generated 93 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:42:59.940414 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-319d144c-d643-44cc-80e8-847b7342bb31 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:00.029212 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:00.041235 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1d48a3-1506-4060-8198-7320d5257597 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:00.085460 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-162205a9-9599-4c7e-94cb-adb3a876484f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:00.086485 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-162205a9-9599-4c7e-94cb-adb3a876484f None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:00.086671 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-319d144c-d643-44cc-80e8-847b7342bb31 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:00.087195 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-162205a9-9599-4c7e-94cb-adb3a876484f None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:00.087686 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-162205a9-9599-4c7e-94cb-adb3a876484f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:00.090074 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:00.098260 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-162205a9-9599-4c7e-94cb-adb3a876484f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:00.101593 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-162205a9-9599-4c7e-94cb-adb3a876484f None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:00.120265 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-162205a9-9599-4c7e-94cb-adb3a876484f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:43:00.120265 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:00.120265 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:43:00.120265 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:43:00.120265 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:43:00.120265 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:43:00.120265 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:43:00.120265 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:43:00.120265 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:43:00.120265 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:43:00.120265 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:43:00.120265 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:43:00.120265 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 00:43:00.120265 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 00:43:00.120265 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 00:43:00.121228 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 00:43:00.121228 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:43:00.121228 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:43:00.121228 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:43:00.121228 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:43:00.121228 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:43:00.121228 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:43:00.121228 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:43:00.121228 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:43:00.121228 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:43:00.121228 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:43:00.121228 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:43:00.121228 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:43:00.121228 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:43:00.121228 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:43:00.121228 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:43:00.121228 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:43:00.122169 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 00:43:00.122169 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 00:43:00.122169 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 00:43:00.122169 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 00:43:00.122169 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 00:43:00.122169 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 00:43:00.122169 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:43:00.122169 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:43:00.129187 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 476/938] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:42:59 2022] GET /identity/v3/projects/service => generated 88 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:43:00.130898 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-162205a9-9599-4c7e-94cb-adb3a876484f None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:00.142094 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-776a54df-a942-4085-80e1-2b97c919afc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:00.142930 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776a54df-a942-4085-80e1-2b97c919afc0 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:00.143305 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776a54df-a942-4085-80e1-2b97c919afc0 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:00.144145 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776a54df-a942-4085-80e1-2b97c919afc0 None admin] PATH_INFO: `/v3/services` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:00.144561 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-776a54df-a942-4085-80e1-2b97c919afc0 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:00.145012 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-776a54df-a942-4085-80e1-2b97c919afc0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:00.147248 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-776a54df-a942-4085-80e1-2b97c919afc0 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:00.157359 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 463/939] 10.222.0.52 () {56 vars in 1122 bytes} [Sun Sep 25 00:42:59 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:43:00.159026 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-776a54df-a942-4085-80e1-2b97c919afc0 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:00.262789 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-162205a9-9599-4c7e-94cb-adb3a876484f None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:00.313420 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe0e99fb-06da-4440-a311-5ce103ec6733 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:00.314279 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0e99fb-06da-4440-a311-5ce103ec6733 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:00.315178 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0e99fb-06da-4440-a311-5ce103ec6733 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:00.315617 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0e99fb-06da-4440-a311-5ce103ec6733 None admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:00.316630 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe0e99fb-06da-4440-a311-5ce103ec6733 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:00.317230 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe0e99fb-06da-4440-a311-5ce103ec6733 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:00.319102 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe0e99fb-06da-4440-a311-5ce103ec6733 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:00.321027 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-776a54df-a942-4085-80e1-2b97c919afc0 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:00.339836 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 477/940] 10.222.0.52 () {56 vars in 1148 bytes} [Sun Sep 25 00:43:00 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:00.340978 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe0e99fb-06da-4440-a311-5ce103ec6733 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:00.357967 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:00.376903 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8eceb1d-bac2-4ac5-b8b6-299ca53fd00e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:00.377631 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8eceb1d-bac2-4ac5-b8b6-299ca53fd00e None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:00.377820 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8eceb1d-bac2-4ac5-b8b6-299ca53fd00e None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:00.378095 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8eceb1d-bac2-4ac5-b8b6-299ca53fd00e None admin] PATH_INFO: `/v3/services` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:00.378896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8eceb1d-bac2-4ac5-b8b6-299ca53fd00e None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:00.379305 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8eceb1d-bac2-4ac5-b8b6-299ca53fd00e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:00.380917 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8eceb1d-bac2-4ac5-b8b6-299ca53fd00e None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:00.386925 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 464/941] 10.222.0.52 () {56 vars in 1122 bytes} [Sun Sep 25 00:43:00 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:43:00.388446 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed044bc-d654-4c7f-a933-4115905c8276 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:00.388732 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed044bc-d654-4c7f-a933-4115905c8276 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:00.389006 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed044bc-d654-4c7f-a933-4115905c8276 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:00.390660 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 465/942] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:43:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:43:00.399747 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed044bc-d654-4c7f-a933-4115905c8276 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:00.430500 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:00.466071 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe0e99fb-06da-4440-a311-5ce103ec6733 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:00.503106 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c57fdf-1545-4efe-af50-b2c556b7a25a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:00.503794 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c57fdf-1545-4efe-af50-b2c556b7a25a None admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:00.503991 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed044bc-d654-4c7f-a933-4115905c8276 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:00.504095 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c57fdf-1545-4efe-af50-b2c556b7a25a None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:00.504319 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c57fdf-1545-4efe-af50-b2c556b7a25a None admin] PATH_INFO: `/v3/projects/9db0fd07fc494203b2ada13f3cd046d3/users/3f20ffd8df994c1db1b1e7248db5b5a3/roles/5d162233989d4e439c441608704dc9b0` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:00.507119 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c57fdf-1545-4efe-af50-b2c556b7a25a None admin] RBAC: Authorizing `identity:create_grant(project_id=9db0fd07fc494203b2ada13f3cd046d3, user_id=3f20ffd8df994c1db1b1e7248db5b5a3, role_id=5d162233989d4e439c441608704dc9b0)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:00.508717 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c57fdf-1545-4efe-af50-b2c556b7a25a None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:00.523596 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 478/943] 10.222.0.52 () {58 vars in 1439 bytes} [Sun Sep 25 00:43:00 2022] PUT /identity/v3/projects/9db0fd07fc494203b2ada13f3cd046d3/users/3f20ffd8df994c1db1b1e7248db5b5a3/roles/5d162233989d4e439c441608704dc9b0 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:43:00.524839 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030f5487-3643-4e07-a96f-4e61c2289043 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:00.525125 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030f5487-3643-4e07-a96f-4e61c2289043 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:00.525416 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030f5487-3643-4e07-a96f-4e61c2289043 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:00.526799 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 479/944] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:43:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:43:00.532353 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fafef76-77e4-4657-a83f-98559311fbed None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:00.532610 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fafef76-77e4-4657-a83f-98559311fbed None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:00.532877 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fafef76-77e4-4657-a83f-98559311fbed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:00.553793 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-8fafef76-77e4-4657-a83f-98559311fbed None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:43:00.555057 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-8fafef76-77e4-4657-a83f-98559311fbed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:43:00.574907 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4ff5f5-2734-4541-a705-275382cc47e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:00.575489 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4ff5f5-2734-4541-a705-275382cc47e9 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:00.575759 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4ff5f5-2734-4541-a705-275382cc47e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:00.576023 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4ff5f5-2734-4541-a705-275382cc47e9 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:00.576763 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e4ff5f5-2734-4541-a705-275382cc47e9 None admin] RBAC: Adding query filter params (interface=public, service_id=23b99b8be13642db92077f78b96e2dc4, region_id=RegionOne) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:00.577131 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e4ff5f5-2734-4541-a705-275382cc47e9 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:00.578649 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e4ff5f5-2734-4541-a705-275382cc47e9 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:00.585549 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 466/945] 10.222.0.52 () {56 vars in 1251 bytes} [Sun Sep 25 00:43:00 2022] GET /identity/v3/endpoints?service_id=23b99b8be13642db92077f78b96e2dc4&interface=public®ion_id=RegionOne => generated 194 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:00.591223 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2393a0-57a7-47e8-8553-c00f4ffa1d4b None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:00.591554 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2393a0-57a7-47e8-8553-c00f4ffa1d4b None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:00.591776 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2393a0-57a7-47e8-8553-c00f4ffa1d4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:00.612494 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-7f2393a0-57a7-47e8-8553-c00f4ffa1d4b None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:43:00.614087 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-7f2393a0-57a7-47e8-8553-c00f4ffa1d4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:43:00.638940 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:00.645191 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 480/946] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:43:00 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:43:00.646617 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fafef76-77e4-4657-a83f-98559311fbed None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:00.725290 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 467/947] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:43:00 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:43:00.785673 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fafef76-77e4-4657-a83f-98559311fbed None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:00.827397 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-334cdb64-9599-4b7e-bc45-bae76f0e42f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:00.827978 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334cdb64-9599-4b7e-bc45-bae76f0e42f7 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:00.828254 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334cdb64-9599-4b7e-bc45-bae76f0e42f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:00.828516 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334cdb64-9599-4b7e-bc45-bae76f0e42f7 None admin] PATH_INFO: `/v3/services` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:00.829127 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-334cdb64-9599-4b7e-bc45-bae76f0e42f7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:00.830637 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-334cdb64-9599-4b7e-bc45-bae76f0e42f7 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:00.838375 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 481/948] 10.222.0.52 () {56 vars in 1087 bytes} [Sun Sep 25 00:43:00 2022] GET /identity/v3/services => generated 1497 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:43:00.924777 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2393a0-57a7-47e8-8553-c00f4ffa1d4b None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:00.953843 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:01.030037 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-334cdb64-9599-4b7e-bc45-bae76f0e42f7 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:01.049046 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2393a0-57a7-47e8-8553-c00f4ffa1d4b None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:01.092213 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6fe6bc1-c186-4bf0-a4bd-0e77ce8b486c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:01.092768 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6fe6bc1-c186-4bf0-a4bd-0e77ce8b486c None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:01.093048 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6fe6bc1-c186-4bf0-a4bd-0e77ce8b486c None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:01.093337 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6fe6bc1-c186-4bf0-a4bd-0e77ce8b486c None admin] PATH_INFO: `/v3/services/compute` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:01.093922 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6fe6bc1-c186-4bf0-a4bd-0e77ce8b486c None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:01.095283 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6fe6bc1-c186-4bf0-a4bd-0e77ce8b486c None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:01.106423 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-b6fe6bc1-c186-4bf0-a4bd-0e77ce8b486c None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 25 00:43:01.106423 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:01.106423 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 00:43:01.106423 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 00:43:01.106423 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 00:43:01.106423 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 00:43:01.106423 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 00:43:01.106423 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:43:01.106423 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 25 00:43:01.106423 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:43:01.106423 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 00:43:01.106423 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:43:01.106423 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:01.106423 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:43:01.106423 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:43:01.106423 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:43:01.106423 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:43:01.107726 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:43:01.107726 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:43:01.107726 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:43:01.107726 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:43:01.107726 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:43:01.107726 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:43:01.107726 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 00:43:01.107726 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 00:43:01.107726 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 00:43:01.107726 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 00:43:01.107726 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:43:01.107726 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:43:01.107726 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:43:01.107726 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:43:01.107726 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:43:01.107726 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:43:01.108723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:43:01.108723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:43:01.108723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:43:01.108723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:43:01.108723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:43:01.108723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:43:01.108723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:43:01.108723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:43:01.108723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:43:01.108723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:43:01.108723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 00:43:01.108723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:43:01.108723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 25 00:43:01.108723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:43:01.108723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 468/949] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:43:00 2022] GET /identity/v3/services/compute => generated 88 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:43:01.113746 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6fe6bc1-c186-4bf0-a4bd-0e77ce8b486c None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:01.116659 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:01.161286 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-334cdb64-9599-4b7e-bc45-bae76f0e42f7 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:01.207841 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9c543f0-a121-4c68-a0ca-96f4f66cd5b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:01.209910 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c543f0-a121-4c68-a0ca-96f4f66cd5b1 None admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:01.210185 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c543f0-a121-4c68-a0ca-96f4f66cd5b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:01.210451 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c543f0-a121-4c68-a0ca-96f4f66cd5b1 None admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:01.211104 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9c543f0-a121-4c68-a0ca-96f4f66cd5b1 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:01.212521 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9c543f0-a121-4c68-a0ca-96f4f66cd5b1 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:01.215208 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: WARNING keystone.server.flask.common [None req-f9c543f0-a121-4c68-a0ca-96f4f66cd5b1 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Sep 25 00:43:01.238845 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.sql.core [None req-f9c543f0-a121-4c68-a0ca-96f4f66cd5b1 None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 25 00:43:01.238845 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Sep 25 00:43:01.238845 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [parameters: {'id': 'c7498c68cabc48f58aa44d7f2b5222f5', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Sep 25 00:43:01.238845 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81386) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 25 00:43:01.244788 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:01.262833 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-f9c543f0-a121-4c68-a0ca-96f4f66cd5b1 None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id c7498c68cabc48f58aa44d7f2b5222f5.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id c7498c68cabc48f58aa44d7f2b5222f5. Sep 25 00:43:01.262833 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:01.262833 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 25 00:43:01.262833 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 25 00:43:01.262833 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 25 00:43:01.262833 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 25 00:43:01.262833 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 25 00:43:01.262833 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application result = self._query(query) Sep 25 00:43:01.262833 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 25 00:43:01.262833 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application conn.query(q) Sep 25 00:43:01.262833 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 25 00:43:01.262833 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 25 00:43:01.262833 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 25 00:43:01.263721 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application result.read() Sep 25 00:43:01.263721 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 25 00:43:01.263721 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 25 00:43:01.263721 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 25 00:43:01.263721 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 25 00:43:01.263721 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 25 00:43:01.263721 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 25 00:43:01.263721 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 25 00:43:01.263721 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 25 00:43:01.263721 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 25 00:43:01.263721 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:43:01.263721 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 25 00:43:01.263721 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:43:01.263721 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:01.263721 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 25 00:43:01.263721 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 25 00:43:01.263721 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Sep 25 00:43:01.263721 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return project_ref.to_dict() Sep 25 00:43:01.264684 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 25 00:43:01.264684 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application next(self.gen) Sep 25 00:43:01.264684 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 25 00:43:01.264684 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application yield resource Sep 25 00:43:01.264684 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 25 00:43:01.264684 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application next(self.gen) Sep 25 00:43:01.264684 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 25 00:43:01.264684 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application self.session.rollback() Sep 25 00:43:01.264684 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 25 00:43:01.264684 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application self.force_reraise() Sep 25 00:43:01.264684 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 25 00:43:01.264684 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise self.value Sep 25 00:43:01.264684 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 25 00:43:01.264684 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 25 00:43:01.264684 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 25 00:43:01.264684 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application session.commit() Sep 25 00:43:01.264684 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 25 00:43:01.265620 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 25 00:43:01.265620 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 25 00:43:01.265620 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application self._prepare_impl() Sep 25 00:43:01.265620 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 25 00:43:01.265620 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application self.session.flush() Sep 25 00:43:01.265620 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 25 00:43:01.265620 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application self._flush(objects) Sep 25 00:43:01.265620 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 25 00:43:01.265620 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 25 00:43:01.265620 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 25 00:43:01.265620 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application compat.raise_( Sep 25 00:43:01.265620 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 25 00:43:01.265620 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception Sep 25 00:43:01.265620 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 25 00:43:01.265620 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application flush_context.execute() Sep 25 00:43:01.265620 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 25 00:43:01.265620 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rec.execute(self) Sep 25 00:43:01.266533 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 25 00:43:01.266533 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 25 00:43:01.266533 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 25 00:43:01.266533 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 25 00:43:01.266533 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Sep 25 00:43:01.266533 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application c = connection._execute_20( Sep 25 00:43:01.266533 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 25 00:43:01.266533 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 25 00:43:01.266533 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 25 00:43:01.266533 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 25 00:43:01.266533 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 25 00:43:01.266533 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 25 00:43:01.266533 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 25 00:43:01.266533 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 25 00:43:01.266533 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 25 00:43:01.269024 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 25 00:43:01.269024 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 25 00:43:01.269024 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception Sep 25 00:43:01.269024 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 25 00:43:01.269024 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 25 00:43:01.269024 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 25 00:43:01.269024 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 25 00:43:01.269024 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 25 00:43:01.269024 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application result = self._query(query) Sep 25 00:43:01.269024 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 25 00:43:01.269024 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application conn.query(q) Sep 25 00:43:01.269024 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 25 00:43:01.269024 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 25 00:43:01.269024 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 25 00:43:01.269024 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application result.read() Sep 25 00:43:01.269024 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 25 00:43:01.269024 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 25 00:43:01.271277 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 25 00:43:01.271277 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 25 00:43:01.271277 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 25 00:43:01.271277 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 25 00:43:01.271277 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 25 00:43:01.271277 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 25 00:43:01.271277 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 25 00:43:01.271277 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Sep 25 00:43:01.271277 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [parameters: {'id': 'c7498c68cabc48f58aa44d7f2b5222f5', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Sep 25 00:43:01.271277 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 25 00:43:01.271277 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:43:01.271277 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 00:43:01.271277 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:43:01.271277 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:01.271277 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Sep 25 00:43:01.273273 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Sep 25 00:43:01.273273 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 25 00:43:01.273273 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 25 00:43:01.273273 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Sep 25 00:43:01.273273 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:43:01.273273 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 00:43:01.273273 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:43:01.273273 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:01.273273 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:43:01.273273 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:43:01.273273 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:43:01.273273 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:43:01.273273 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:43:01.273273 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:43:01.273273 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:43:01.273273 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:43:01.273273 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:43:01.275375 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:43:01.275375 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Sep 25 00:43:01.275375 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Sep 25 00:43:01.275375 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:43:01.275375 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:43:01.275375 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Sep 25 00:43:01.275375 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.Conflict( Sep 25 00:43:01.275375 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id c7498c68cabc48f58aa44d7f2b5222f5. Sep 25 00:43:01.275375 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:43:01.278073 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6fe6bc1-c186-4bf0-a4bd-0e77ce8b486c None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:01.278528 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 482/950] 10.222.0.52 () {60 vars in 1140 bytes} [Sun Sep 25 00:43:01 2022] POST /identity/v3/projects => generated 262 bytes in 249 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:43:01.283673 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9c543f0-a121-4c68-a0ca-96f4f66cd5b1 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:01.316672 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b43190-e1b9-4fa8-8331-ea92288c0bb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:01.318458 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b43190-e1b9-4fa8-8331-ea92288c0bb9 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:01.319371 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b43190-e1b9-4fa8-8331-ea92288c0bb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:01.320131 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b43190-e1b9-4fa8-8331-ea92288c0bb9 None admin] PATH_INFO: `/v3/services` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:01.321226 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61b43190-e1b9-4fa8-8331-ea92288c0bb9 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:01.321755 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61b43190-e1b9-4fa8-8331-ea92288c0bb9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:01.323987 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61b43190-e1b9-4fa8-8331-ea92288c0bb9 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:01.332704 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 469/951] 10.222.0.52 () {56 vars in 1112 bytes} [Sun Sep 25 00:43:01 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:43:01.339196 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b43190-e1b9-4fa8-8331-ea92288c0bb9 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:01.417526 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9c543f0-a121-4c68-a0ca-96f4f66cd5b1 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:01.440619 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:01.457924 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f86adfa-f44e-4030-814b-dc560aa80053 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:01.458870 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f86adfa-f44e-4030-814b-dc560aa80053 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:01.458983 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f86adfa-f44e-4030-814b-dc560aa80053 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:01.459234 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f86adfa-f44e-4030-814b-dc560aa80053 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:01.467054 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f86adfa-f44e-4030-814b-dc560aa80053 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:01.468763 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f86adfa-f44e-4030-814b-dc560aa80053 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:01.484619 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-2f86adfa-f44e-4030-814b-dc560aa80053 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:43:01.484619 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:01.484619 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:43:01.484619 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:43:01.484619 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:43:01.484619 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:43:01.484619 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:43:01.484619 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:43:01.484619 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:43:01.484619 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:43:01.484619 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:43:01.484619 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:43:01.484619 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 00:43:01.484619 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 00:43:01.484619 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 00:43:01.485798 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 00:43:01.485798 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:43:01.485798 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:43:01.485798 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:43:01.485798 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:43:01.485798 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:43:01.485798 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:43:01.485798 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:43:01.485798 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:43:01.485798 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:43:01.485798 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:43:01.485798 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:43:01.485798 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:43:01.485798 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:43:01.485798 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:43:01.485798 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:43:01.485798 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:43:01.486751 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 00:43:01.486751 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 00:43:01.486751 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 00:43:01.486751 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 00:43:01.486751 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 00:43:01.486751 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 00:43:01.486751 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:43:01.486751 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:43:01.488404 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 483/952] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:43:01 2022] GET /identity/v3/projects/service => generated 88 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:43:01.490314 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f86adfa-f44e-4030-814b-dc560aa80053 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:01.493890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b43190-e1b9-4fa8-8331-ea92288c0bb9 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:01.541462 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5274ead6-2a80-4dea-ab58-21cab4dcb355 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:01.542511 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5274ead6-2a80-4dea-ab58-21cab4dcb355 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:01.543059 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5274ead6-2a80-4dea-ab58-21cab4dcb355 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:01.543542 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5274ead6-2a80-4dea-ab58-21cab4dcb355 None admin] PATH_INFO: `/v3/services` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:01.544373 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5274ead6-2a80-4dea-ab58-21cab4dcb355 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:01.544867 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5274ead6-2a80-4dea-ab58-21cab4dcb355 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:01.546635 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5274ead6-2a80-4dea-ab58-21cab4dcb355 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:01.552281 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 470/953] 10.222.0.52 () {56 vars in 1112 bytes} [Sun Sep 25 00:43:01 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:01.607765 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f86adfa-f44e-4030-814b-dc560aa80053 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:01.649346 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f526bf3-838b-454a-918f-fdadd63b759a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:01.649973 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f526bf3-838b-454a-918f-fdadd63b759a None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:01.650249 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f526bf3-838b-454a-918f-fdadd63b759a None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:01.650560 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f526bf3-838b-454a-918f-fdadd63b759a None admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:01.651330 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f526bf3-838b-454a-918f-fdadd63b759a None admin] RBAC: Adding query filter params (name=service) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:01.651631 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f526bf3-838b-454a-918f-fdadd63b759a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:01.653252 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f526bf3-838b-454a-918f-fdadd63b759a None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:01.681547 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 484/954] 10.222.0.52 () {56 vars in 1112 bytes} [Sun Sep 25 00:43:01 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:02.871472 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432f7de1-58e4-47c3-98f3-8063710528fd None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:02.871966 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432f7de1-58e4-47c3-98f3-8063710528fd None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:02.871966 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432f7de1-58e4-47c3-98f3-8063710528fd None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:02.873517 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 471/955] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:43:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:43:02.878766 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12c2f86-84f3-4e48-ae35-fa3a30ba796b None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:02.879177 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12c2f86-84f3-4e48-ae35-fa3a30ba796b None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:02.879276 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12c2f86-84f3-4e48-ae35-fa3a30ba796b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:02.899171 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-a12c2f86-84f3-4e48-ae35-fa3a30ba796b None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:43:02.900668 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-a12c2f86-84f3-4e48-ae35-fa3a30ba796b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:43:02.937767 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:02.973110 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 485/956] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:43:02 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:43:03.147433 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b75941-759d-4f62-9612-27b158d4e185 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:03.148059 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b75941-759d-4f62-9612-27b158d4e185 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:03.148538 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b75941-759d-4f62-9612-27b158d4e185 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:03.150371 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 472/957] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:43:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:43:03.156163 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-291053f6-7661-46e7-92fa-a794fedd9fa2 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:03.156722 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-291053f6-7661-46e7-92fa-a794fedd9fa2 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:03.157214 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-291053f6-7661-46e7-92fa-a794fedd9fa2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:03.176097 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-291053f6-7661-46e7-92fa-a794fedd9fa2 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:43:03.178254 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-291053f6-7661-46e7-92fa-a794fedd9fa2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:43:03.246861 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 486/958] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:43:03 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:43:03.265568 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b75941-759d-4f62-9612-27b158d4e185 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:03.286061 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:03.379595 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b75941-759d-4f62-9612-27b158d4e185 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:03.417017 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1496edbf-6aa9-4857-a606-d31fe9851a7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:03.417586 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1496edbf-6aa9-4857-a606-d31fe9851a7d None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:03.417858 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1496edbf-6aa9-4857-a606-d31fe9851a7d None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:03.418118 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1496edbf-6aa9-4857-a606-d31fe9851a7d None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:03.429927 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1496edbf-6aa9-4857-a606-d31fe9851a7d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:43:03.434720 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-1496edbf-6aa9-4857-a606-d31fe9851a7d None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 25 00:43:03.434720 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:03.434720 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:43:03.434720 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:43:03.434720 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:43:03.434720 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:43:03.434720 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:43:03.434720 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:43:03.434720 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:43:03.434720 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:43:03.434720 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:43:03.434720 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:43:03.434720 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:43:03.434720 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:43:03.434720 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:43:03.434720 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:43:03.435687 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:43:03.435687 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:43:03.435687 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:43:03.435687 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:43:03.435687 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:43:03.435687 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:43:03.435687 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:43:03.435687 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:43:03.435687 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:43:03.435687 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:43:03.435687 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:43:03.435687 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:43:03.435687 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:43:03.435687 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:43:03.435687 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:43:03.435687 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:43:03.436616 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:43:03.436616 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:43:03.436616 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:43:03.436616 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:43:03.436616 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:43:03.436616 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:43:03.436616 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:43:03.436616 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:43:03.436616 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:43:03.436616 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:43:03.436616 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 25 00:43:03.436616 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:43:03.436616 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 473/959] 10.222.0.52 () {56 vars in 1120 bytes} [Sun Sep 25 00:43:03 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:43:03.441817 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-291053f6-7661-46e7-92fa-a794fedd9fa2 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:03.578630 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1496edbf-6aa9-4857-a606-d31fe9851a7d None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:03.612253 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-291053f6-7661-46e7-92fa-a794fedd9fa2 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:03.617652 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:03.669678 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:03.690458 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd8ab366-65f0-4f32-b2d0-0d6be1b6fbef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:03.692076 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8ab366-65f0-4f32-b2d0-0d6be1b6fbef None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:03.692562 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8ab366-65f0-4f32-b2d0-0d6be1b6fbef None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:03.694012 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8ab366-65f0-4f32-b2d0-0d6be1b6fbef None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:03.695322 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd8ab366-65f0-4f32-b2d0-0d6be1b6fbef None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:03.695838 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd8ab366-65f0-4f32-b2d0-0d6be1b6fbef None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:03.698494 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd8ab366-65f0-4f32-b2d0-0d6be1b6fbef None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:03.703747 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1496edbf-6aa9-4857-a606-d31fe9851a7d None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:03.721345 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 487/960] 10.222.0.52 () {56 vars in 1115 bytes} [Sun Sep 25 00:43:03 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:43:03.725522 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd8ab366-65f0-4f32-b2d0-0d6be1b6fbef None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:03.770769 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09be8e49-51f8-4f07-a171-7999a25136d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:03.771916 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09be8e49-51f8-4f07-a171-7999a25136d7 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:03.772412 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09be8e49-51f8-4f07-a171-7999a25136d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:03.772895 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09be8e49-51f8-4f07-a171-7999a25136d7 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:03.773996 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09be8e49-51f8-4f07-a171-7999a25136d7 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:03.777812 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09be8e49-51f8-4f07-a171-7999a25136d7 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:03.799135 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-09be8e49-51f8-4f07-a171-7999a25136d7 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 25 00:43:03.799135 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:03.799135 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 00:43:03.799135 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 00:43:03.799135 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 00:43:03.799135 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 00:43:03.799135 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 00:43:03.799135 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:43:03.799135 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 25 00:43:03.799135 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:43:03.799135 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 00:43:03.799135 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:43:03.799135 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:03.799135 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:43:03.799135 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:43:03.799135 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:43:03.799135 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:43:03.800327 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:43:03.800327 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:43:03.800327 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:43:03.800327 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:43:03.800327 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:43:03.800327 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:43:03.800327 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 00:43:03.800327 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 00:43:03.800327 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 00:43:03.800327 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 00:43:03.800327 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:43:03.800327 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:43:03.800327 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:43:03.800327 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:43:03.800327 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:43:03.800327 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:43:03.801197 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:43:03.801197 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:43:03.801197 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:43:03.801197 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:43:03.801197 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:43:03.801197 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:43:03.801197 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:43:03.801197 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:43:03.801197 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:43:03.801197 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:43:03.801197 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 00:43:03.801197 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:43:03.801197 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 25 00:43:03.801197 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:43:03.801197 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 474/961] 10.222.0.52 () {56 vars in 1126 bytes} [Sun Sep 25 00:43:03 2022] GET /identity/v3/services/object-store => generated 93 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:43:03.803999 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10065400-74b1-4d52-8f7d-c81ee011baca None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:03.804507 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10065400-74b1-4d52-8f7d-c81ee011baca None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:03.805044 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10065400-74b1-4d52-8f7d-c81ee011baca None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:03.807659 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 475/962] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:43:03 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:43:03.811513 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10065400-74b1-4d52-8f7d-c81ee011baca None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:03.852501 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd8ab366-65f0-4f32-b2d0-0d6be1b6fbef None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:03.888554 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-871e28c3-c0f3-49a9-b631-2d07da714765 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:03.889202 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871e28c3-c0f3-49a9-b631-2d07da714765 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:03.889476 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871e28c3-c0f3-49a9-b631-2d07da714765 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:03.889748 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871e28c3-c0f3-49a9-b631-2d07da714765 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:03.902308 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-871e28c3-c0f3-49a9-b631-2d07da714765 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:03.904203 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-871e28c3-c0f3-49a9-b631-2d07da714765 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:03.920993 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-871e28c3-c0f3-49a9-b631-2d07da714765 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:43:03.920993 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:03.920993 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:43:03.920993 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:43:03.920993 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:43:03.920993 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:43:03.920993 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:43:03.920993 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:43:03.920993 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:43:03.920993 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:43:03.920993 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:43:03.920993 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:43:03.920993 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:43:03.920993 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:43:03.920993 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:43:03.920993 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:43:03.922078 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:43:03.922078 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:43:03.922078 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:43:03.922078 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:43:03.922078 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:43:03.922078 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:43:03.922078 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:43:03.922078 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:43:03.922078 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:43:03.922078 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:43:03.922078 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:43:03.922078 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:43:03.922078 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:43:03.922078 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:43:03.922078 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:43:03.922078 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:43:03.922078 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:43:03.922991 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:43:03.922991 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:43:03.922991 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:43:03.922991 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 488/963] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:43:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:43:03.924353 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3120e995-00b0-4809-a92a-6a781860c598 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:03.924353 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3120e995-00b0-4809-a92a-6a781860c598 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:03.924353 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3120e995-00b0-4809-a92a-6a781860c598 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:03.943314 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10065400-74b1-4d52-8f7d-c81ee011baca None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:03.947233 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-3120e995-00b0-4809-a92a-6a781860c598 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:43:03.948674 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-3120e995-00b0-4809-a92a-6a781860c598 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:43:03.960979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:03.983703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-264af975-7614-4544-8332-0eed25220d4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:03.984375 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-264af975-7614-4544-8332-0eed25220d4e None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:03.984645 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-264af975-7614-4544-8332-0eed25220d4e None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:03.984916 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-264af975-7614-4544-8332-0eed25220d4e None admin] PATH_INFO: `/v3/services` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:03.985626 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-264af975-7614-4544-8332-0eed25220d4e None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:03.985987 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-264af975-7614-4544-8332-0eed25220d4e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:03.987283 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-264af975-7614-4544-8332-0eed25220d4e None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:03.993191 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 476/964] 10.222.0.52 () {56 vars in 1122 bytes} [Sun Sep 25 00:43:03 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:43:03.994036 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-264af975-7614-4544-8332-0eed25220d4e None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:04.000036 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:04.036070 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 489/965] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:43:03 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:43:04.037361 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3120e995-00b0-4809-a92a-6a781860c598 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:04.108085 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-264af975-7614-4544-8332-0eed25220d4e None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:04.142660 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d0ff0d-0676-452a-88ac-763fa837c72d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:04.143262 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d0ff0d-0676-452a-88ac-763fa837c72d None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:04.143537 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d0ff0d-0676-452a-88ac-763fa837c72d None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:04.143805 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d0ff0d-0676-452a-88ac-763fa837c72d None admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:04.144437 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d0ff0d-0676-452a-88ac-763fa837c72d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:04.144740 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d0ff0d-0676-452a-88ac-763fa837c72d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:04.146691 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d0ff0d-0676-452a-88ac-763fa837c72d None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:04.149340 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3120e995-00b0-4809-a92a-6a781860c598 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:04.163443 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 477/966] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:43:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:04.163443 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d35b2dc-823e-4bbd-9b75-ef525dc643e3 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:04.163443 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d35b2dc-823e-4bbd-9b75-ef525dc643e3 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:04.163443 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d35b2dc-823e-4bbd-9b75-ef525dc643e3 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:04.163443 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 478/967] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:43:04 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:43:04.166887 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d35b2dc-823e-4bbd-9b75-ef525dc643e3 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:04.192030 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaafd951-eab0-427f-a5b1-91f6b8c657ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:04.192544 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaafd951-eab0-427f-a5b1-91f6b8c657ac None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:04.193444 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaafd951-eab0-427f-a5b1-91f6b8c657ac None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:04.193873 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaafd951-eab0-427f-a5b1-91f6b8c657ac None admin] PATH_INFO: `/v3/services` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:04.194706 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaafd951-eab0-427f-a5b1-91f6b8c657ac None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:04.195008 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaafd951-eab0-427f-a5b1-91f6b8c657ac None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:04.196879 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaafd951-eab0-427f-a5b1-91f6b8c657ac None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:04.206792 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 490/968] 10.222.0.52 () {56 vars in 1122 bytes} [Sun Sep 25 00:43:04 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:04.206792 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443aed58-00d4-4187-a548-374c448952c2 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:04.206792 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443aed58-00d4-4187-a548-374c448952c2 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:04.206792 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443aed58-00d4-4187-a548-374c448952c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:04.222488 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-443aed58-00d4-4187-a548-374c448952c2 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:43:04.223656 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-443aed58-00d4-4187-a548-374c448952c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:43:04.226563 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:04.241672 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:04.284365 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d35b2dc-823e-4bbd-9b75-ef525dc643e3 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:04.301301 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 491/969] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:43:04 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:43:04.301301 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-443aed58-00d4-4187-a548-374c448952c2 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:04.337900 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef50712a-4408-44db-8a77-374c4feacfd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:04.338626 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef50712a-4408-44db-8a77-374c4feacfd5 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:04.338903 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef50712a-4408-44db-8a77-374c4feacfd5 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:04.339172 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef50712a-4408-44db-8a77-374c4feacfd5 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:04.349703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef50712a-4408-44db-8a77-374c4feacfd5 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:04.351304 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef50712a-4408-44db-8a77-374c4feacfd5 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:04.363537 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-ef50712a-4408-44db-8a77-374c4feacfd5 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 25 00:43:04.363537 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:04.363537 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:43:04.363537 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:43:04.363537 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:43:04.363537 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:43:04.363537 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:43:04.363537 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:43:04.363537 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:43:04.363537 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:43:04.363537 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:43:04.363537 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:43:04.363537 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 00:43:04.363537 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 00:43:04.363537 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 00:43:04.363537 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 00:43:04.364575 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:43:04.364575 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:43:04.364575 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 00:43:04.364575 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:43:04.364575 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 00:43:04.364575 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:43:04.364575 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:43:04.364575 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:43:04.364575 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:43:04.364575 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:43:04.364575 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:43:04.364575 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:43:04.364575 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:43:04.364575 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:43:04.364575 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:43:04.364575 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:43:04.364575 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:43:04.364575 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:43:04.365869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:43:04.365869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:43:04.365869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 00:43:04.365869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 00:43:04.365869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 00:43:04.365869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 00:43:04.365869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 00:43:04.365869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 00:43:04.365869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 25 00:43:04.365869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:43:04.365869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 479/970] 10.222.0.52 () {56 vars in 1117 bytes} [Sun Sep 25 00:43:04 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:43:04.366570 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef50712a-4408-44db-8a77-374c4feacfd5 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:04.420637 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-443aed58-00d4-4187-a548-374c448952c2 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:04.461078 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d213bdd-5cd8-4839-9965-446c553a8028 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:04.461628 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d213bdd-5cd8-4839-9965-446c553a8028 None admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:04.461911 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d213bdd-5cd8-4839-9965-446c553a8028 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:04.462176 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d213bdd-5cd8-4839-9965-446c553a8028 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:04.462838 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d213bdd-5cd8-4839-9965-446c553a8028 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:04.464374 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d213bdd-5cd8-4839-9965-446c553a8028 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:04.473601 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:04.476093 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef50712a-4408-44db-8a77-374c4feacfd5 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:04.492883 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 492/971] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:43:04 2022] POST /identity/v3/endpoints => generated 349 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:43:04.494453 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d213bdd-5cd8-4839-9965-446c553a8028 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:04.535319 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b566f2ed-00ef-4b87-bca4-15322edaeb9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:04.536231 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b566f2ed-00ef-4b87-bca4-15322edaeb9b None admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:04.536649 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b566f2ed-00ef-4b87-bca4-15322edaeb9b None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:04.536990 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b566f2ed-00ef-4b87-bca4-15322edaeb9b None admin] PATH_INFO: `/v3/services` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:04.538021 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b566f2ed-00ef-4b87-bca4-15322edaeb9b None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:04.540564 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:04.541106 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b566f2ed-00ef-4b87-bca4-15322edaeb9b None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:04.563925 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 480/972] 10.222.0.52 () {60 vars in 1141 bytes} [Sun Sep 25 00:43:04 2022] POST /identity/v3/services => generated 242 bytes in 198 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:43:04.576208 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b566f2ed-00ef-4b87-bca4-15322edaeb9b None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:04.661081 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d213bdd-5cd8-4839-9965-446c553a8028 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:04.720179 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b04bde-ce05-450a-bd84-c1d28d6c1f65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:04.720732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b04bde-ce05-450a-bd84-c1d28d6c1f65 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:04.721164 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b04bde-ce05-450a-bd84-c1d28d6c1f65 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:04.721484 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b04bde-ce05-450a-bd84-c1d28d6c1f65 None admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:04.722244 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b04bde-ce05-450a-bd84-c1d28d6c1f65 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:04.722550 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b04bde-ce05-450a-bd84-c1d28d6c1f65 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:04.724036 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b04bde-ce05-450a-bd84-c1d28d6c1f65 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:04.751614 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 493/973] 10.222.0.52 () {56 vars in 1149 bytes} [Sun Sep 25 00:43:04 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:04.751937 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b566f2ed-00ef-4b87-bca4-15322edaeb9b None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:04.752841 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b04bde-ce05-450a-bd84-c1d28d6c1f65 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:04.793586 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3040276-ad10-495f-86f8-1ebc0c0da386 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:04.794229 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3040276-ad10-495f-86f8-1ebc0c0da386 None admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:04.794514 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3040276-ad10-495f-86f8-1ebc0c0da386 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:04.794776 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3040276-ad10-495f-86f8-1ebc0c0da386 None admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:04.796182 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3040276-ad10-495f-86f8-1ebc0c0da386 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:04.797791 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3040276-ad10-495f-86f8-1ebc0c0da386 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:04.800378 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: WARNING keystone.server.flask.common [None req-c3040276-ad10-495f-86f8-1ebc0c0da386 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Sep 25 00:43:04.815028 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:04.817326 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.sql.core [None req-c3040276-ad10-495f-86f8-1ebc0c0da386 None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 25 00:43:04.817326 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Sep 25 00:43:04.817326 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [parameters: {'user_id': 'a7dc06423c5b41938a0310515f280dac', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Sep 25 00:43:04.817326 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81387) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 25 00:43:04.825426 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-c3040276-ad10-495f-86f8-1ebc0c0da386 None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Sep 25 00:43:04.825426 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:04.825426 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 25 00:43:04.825426 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 25 00:43:04.825426 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 25 00:43:04.825426 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 25 00:43:04.825426 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 25 00:43:04.825426 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application result = self._query(query) Sep 25 00:43:04.825426 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 25 00:43:04.825426 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application conn.query(q) Sep 25 00:43:04.825426 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 25 00:43:04.825426 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 25 00:43:04.825426 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 25 00:43:04.825426 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application result.read() Sep 25 00:43:04.825426 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 25 00:43:04.826438 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 25 00:43:04.826438 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 25 00:43:04.826438 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 25 00:43:04.826438 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 25 00:43:04.826438 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 25 00:43:04.826438 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 25 00:43:04.826438 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 25 00:43:04.826438 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 25 00:43:04.826438 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:43:04.826438 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 25 00:43:04.826438 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:43:04.826438 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:04.826438 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 25 00:43:04.826438 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 25 00:43:04.826438 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Sep 25 00:43:04.826438 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Sep 25 00:43:04.826438 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 25 00:43:04.826438 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application next(self.gen) Sep 25 00:43:04.827505 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 25 00:43:04.827505 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application yield resource Sep 25 00:43:04.827505 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 25 00:43:04.827505 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application next(self.gen) Sep 25 00:43:04.827505 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 25 00:43:04.827505 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application self.session.rollback() Sep 25 00:43:04.827505 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 25 00:43:04.827505 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application self.force_reraise() Sep 25 00:43:04.827505 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 25 00:43:04.827505 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise self.value Sep 25 00:43:04.827505 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 25 00:43:04.827505 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 25 00:43:04.827505 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 25 00:43:04.827505 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application session.commit() Sep 25 00:43:04.827505 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 25 00:43:04.827505 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 25 00:43:04.827505 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 25 00:43:04.828440 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application self._prepare_impl() Sep 25 00:43:04.828440 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 25 00:43:04.828440 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application self.session.flush() Sep 25 00:43:04.828440 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 25 00:43:04.828440 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application self._flush(objects) Sep 25 00:43:04.828440 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 25 00:43:04.828440 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 25 00:43:04.828440 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 25 00:43:04.828440 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application compat.raise_( Sep 25 00:43:04.828440 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 25 00:43:04.828440 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception Sep 25 00:43:04.828440 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 25 00:43:04.828440 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application flush_context.execute() Sep 25 00:43:04.828440 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 25 00:43:04.828440 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rec.execute(self) Sep 25 00:43:04.828440 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 25 00:43:04.828440 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 25 00:43:04.829467 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 25 00:43:04.829467 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 25 00:43:04.829467 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Sep 25 00:43:04.829467 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application result = connection._execute_20( Sep 25 00:43:04.829467 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 25 00:43:04.829467 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 25 00:43:04.829467 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 25 00:43:04.829467 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 25 00:43:04.829467 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 25 00:43:04.829467 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 25 00:43:04.829467 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 25 00:43:04.829467 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 25 00:43:04.829467 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 25 00:43:04.829467 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 25 00:43:04.829467 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 25 00:43:04.829467 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception Sep 25 00:43:04.830374 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 25 00:43:04.830374 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 25 00:43:04.830374 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 25 00:43:04.830374 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 25 00:43:04.830374 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 25 00:43:04.830374 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application result = self._query(query) Sep 25 00:43:04.830374 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 25 00:43:04.830374 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application conn.query(q) Sep 25 00:43:04.830374 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 25 00:43:04.830374 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 25 00:43:04.830374 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 25 00:43:04.830374 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application result.read() Sep 25 00:43:04.830374 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 25 00:43:04.830374 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 25 00:43:04.830374 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 25 00:43:04.830374 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 25 00:43:04.831244 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 25 00:43:04.831244 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 25 00:43:04.831244 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 25 00:43:04.831244 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 25 00:43:04.831244 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 25 00:43:04.831244 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Sep 25 00:43:04.831244 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [parameters: {'user_id': 'a7dc06423c5b41938a0310515f280dac', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Sep 25 00:43:04.831244 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 25 00:43:04.831244 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:43:04.831244 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 00:43:04.831244 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:43:04.831244 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:04.831244 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:43:04.831244 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:43:04.831244 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:43:04.832059 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:43:04.832059 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:43:04.832059 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:43:04.832059 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:43:04.832059 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:43:04.832059 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:43:04.832059 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:43:04.832059 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Sep 25 00:43:04.832059 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Sep 25 00:43:04.832059 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:43:04.832059 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:43:04.832059 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 00:43:04.832059 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:43:04.832059 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 00:43:04.832059 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:43:04.832059 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Sep 25 00:43:04.832059 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Sep 25 00:43:04.833015 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Sep 25 00:43:04.833015 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Sep 25 00:43:04.833015 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 25 00:43:04.833015 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 25 00:43:04.833015 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Sep 25 00:43:04.833015 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:43:04.833015 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 481/974] 10.222.0.52 () {60 vars in 1131 bytes} [Sun Sep 25 00:43:04 2022] POST /identity/v3/users => generated 160 bytes in 251 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:43:04.837292 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3040276-ad10-495f-86f8-1ebc0c0da386 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:04.884281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b04bde-ce05-450a-bd84-c1d28d6c1f65 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:04.900916 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:04.926520 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37de5010-f592-40d6-98bd-316ece9329c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:04.928018 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37de5010-f592-40d6-98bd-316ece9329c9 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:04.928576 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37de5010-f592-40d6-98bd-316ece9329c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:04.929227 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37de5010-f592-40d6-98bd-316ece9329c9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:04.997738 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3040276-ad10-495f-86f8-1ebc0c0da386 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:05.058254 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15188fc1-93f4-4c5b-bcd7-0a8ae67043ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:05.059626 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15188fc1-93f4-4c5b-bcd7-0a8ae67043ce None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:05.060359 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15188fc1-93f4-4c5b-bcd7-0a8ae67043ce None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:05.061056 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15188fc1-93f4-4c5b-bcd7-0a8ae67043ce None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:05.078092 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15188fc1-93f4-4c5b-bcd7-0a8ae67043ce None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:05.080940 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15188fc1-93f4-4c5b-bcd7-0a8ae67043ce None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:05.084422 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37de5010-f592-40d6-98bd-316ece9329c9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:05.087303 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37de5010-f592-40d6-98bd-316ece9329c9 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:05.101975 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-15188fc1-93f4-4c5b-bcd7-0a8ae67043ce None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 25 00:43:05.101975 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:05.101975 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:43:05.101975 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:43:05.101975 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:43:05.101975 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:43:05.101975 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:43:05.101975 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:43:05.101975 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:43:05.101975 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:43:05.101975 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:43:05.101975 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:43:05.101975 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 00:43:05.101975 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 00:43:05.101975 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 00:43:05.101975 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 00:43:05.103830 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:43:05.103830 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:43:05.103830 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 00:43:05.103830 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:43:05.103830 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 00:43:05.103830 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:43:05.103830 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:43:05.103830 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:43:05.103830 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:43:05.103830 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:43:05.103830 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:43:05.103830 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:43:05.103830 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:43:05.103830 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:43:05.103830 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:43:05.103830 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:43:05.103830 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:43:05.103830 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:43:05.105558 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:43:05.105558 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:43:05.105558 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 00:43:05.105558 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 00:43:05.105558 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 00:43:05.105558 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 00:43:05.105558 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 00:43:05.105558 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 00:43:05.105558 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 25 00:43:05.105558 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:43:05.109005 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 482/975] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:43:04 2022] GET /identity/v3/users/cinder => generated 84 bytes in 272 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:43:05.118990 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15188fc1-93f4-4c5b-bcd7-0a8ae67043ce None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:05.172941 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:05.245735 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 494/976] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:43:04 2022] GET /identity/v3/auth/tokens => generated 1570 bytes in 493 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 25 00:43:05.246865 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37de5010-f592-40d6-98bd-316ece9329c9 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:05.287530 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15188fc1-93f4-4c5b-bcd7-0a8ae67043ce None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:05.303596 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:05.337220 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcbc9621-ee48-40dd-9325-f398cd105566 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:05.338406 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcbc9621-ee48-40dd-9325-f398cd105566 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:05.338859 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcbc9621-ee48-40dd-9325-f398cd105566 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:05.339284 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcbc9621-ee48-40dd-9325-f398cd105566 None admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:05.340132 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcbc9621-ee48-40dd-9325-f398cd105566 None admin] RBAC: Adding query filter params (name=cinder) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:05.340843 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcbc9621-ee48-40dd-9325-f398cd105566 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:05.343078 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcbc9621-ee48-40dd-9325-f398cd105566 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:05.382153 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37de5010-f592-40d6-98bd-316ece9329c9 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:05.393289 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 483/977] 10.222.0.52 () {56 vars in 1101 bytes} [Sun Sep 25 00:43:05 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:43:05.424565 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e89760d9-1cd1-4185-89a2-6297268e97ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:05.425392 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89760d9-1cd1-4185-89a2-6297268e97ae None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:05.425793 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89760d9-1cd1-4185-89a2-6297268e97ae None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:05.426168 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89760d9-1cd1-4185-89a2-6297268e97ae None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:05.438836 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e89760d9-1cd1-4185-89a2-6297268e97ae None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:05.440468 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e89760d9-1cd1-4185-89a2-6297268e97ae None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:05.451072 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:05.458975 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-e89760d9-1cd1-4185-89a2-6297268e97ae None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:43:05.458975 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:05.458975 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:43:05.458975 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:43:05.458975 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:43:05.458975 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:43:05.458975 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:43:05.458975 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:43:05.458975 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:43:05.458975 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:43:05.458975 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:43:05.458975 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:43:05.458975 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:43:05.458975 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:43:05.458975 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:43:05.458975 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:43:05.460222 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:43:05.460222 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:43:05.460222 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:43:05.460222 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:43:05.460222 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:43:05.460222 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:43:05.460222 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:43:05.460222 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:43:05.460222 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:43:05.460222 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:43:05.460222 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:43:05.460222 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:43:05.460222 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:43:05.460222 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:43:05.460222 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:43:05.460222 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:43:05.460222 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:43:05.461443 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:43:05.461443 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:43:05.461443 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:43:05.467526 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 495/978] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:43:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:43:05.469844 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcbc9621-ee48-40dd-9325-f398cd105566 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:05.599669 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcbc9621-ee48-40dd-9325-f398cd105566 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:05.644580 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b0cb26-e745-4ef7-ac3e-c4d2f2fb707c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:05.645467 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b0cb26-e745-4ef7-ac3e-c4d2f2fb707c None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:05.646422 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b0cb26-e745-4ef7-ac3e-c4d2f2fb707c None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:05.647011 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b0cb26-e745-4ef7-ac3e-c4d2f2fb707c None admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:05.647794 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04b0cb26-e745-4ef7-ac3e-c4d2f2fb707c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:05.648271 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04b0cb26-e745-4ef7-ac3e-c4d2f2fb707c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:05.649890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04b0cb26-e745-4ef7-ac3e-c4d2f2fb707c None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:05.663945 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 484/979] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:43:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:05.669453 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e89760d9-1cd1-4185-89a2-6297268e97ae None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:05.787150 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e89760d9-1cd1-4185-89a2-6297268e97ae None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:05.820996 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb1d11a-3971-4a1e-a827-d6c22dfc1d60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:05.821651 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb1d11a-3971-4a1e-a827-d6c22dfc1d60 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:05.821924 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb1d11a-3971-4a1e-a827-d6c22dfc1d60 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:05.822191 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb1d11a-3971-4a1e-a827-d6c22dfc1d60 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:05.828970 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb1d11a-3971-4a1e-a827-d6c22dfc1d60 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:05.830424 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb1d11a-3971-4a1e-a827-d6c22dfc1d60 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:05.842022 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-4bb1d11a-3971-4a1e-a827-d6c22dfc1d60 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:43:05.842022 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:05.842022 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:43:05.842022 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:43:05.842022 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:43:05.842022 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:43:05.842022 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:43:05.842022 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:43:05.842022 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:43:05.842022 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:43:05.842022 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:43:05.842022 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:43:05.842022 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 00:43:05.842022 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 00:43:05.842022 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 00:43:05.843728 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 00:43:05.843728 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:43:05.843728 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:43:05.843728 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:43:05.843728 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:43:05.843728 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:43:05.843728 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:43:05.843728 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:43:05.843728 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:43:05.843728 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:43:05.843728 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:43:05.843728 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:43:05.843728 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:43:05.843728 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:43:05.843728 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:43:05.843728 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:43:05.843728 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:43:05.845422 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 00:43:05.845422 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 00:43:05.845422 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 00:43:05.845422 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 00:43:05.845422 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 00:43:05.845422 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 00:43:05.845422 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:43:05.845422 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:43:05.845422 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 496/980] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:43:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:43:05.850749 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b0cb26-e745-4ef7-ac3e-c4d2f2fb707c None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:05.877454 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:05.965170 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b0cb26-e745-4ef7-ac3e-c4d2f2fb707c None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:06.001178 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d204999-fb02-4efe-9ba0-47aeb2be7be1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:06.001873 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d204999-fb02-4efe-9ba0-47aeb2be7be1 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:06.002099 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d204999-fb02-4efe-9ba0-47aeb2be7be1 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:06.002321 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d204999-fb02-4efe-9ba0-47aeb2be7be1 None admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:06.003007 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d204999-fb02-4efe-9ba0-47aeb2be7be1 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:06.003306 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d204999-fb02-4efe-9ba0-47aeb2be7be1 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:06.004713 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d204999-fb02-4efe-9ba0-47aeb2be7be1 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:06.025294 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 485/981] 10.222.0.52 () {56 vars in 1148 bytes} [Sun Sep 25 00:43:05 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:43:06.027644 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb1d11a-3971-4a1e-a827-d6c22dfc1d60 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:06.093550 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:06.140235 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb1d11a-3971-4a1e-a827-d6c22dfc1d60 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:06.176496 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1368abc-89ae-4b1f-bae9-6a4bbdb91c69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:06.177644 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1368abc-89ae-4b1f-bae9-6a4bbdb91c69 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:06.178023 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1368abc-89ae-4b1f-bae9-6a4bbdb91c69 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:06.178525 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1368abc-89ae-4b1f-bae9-6a4bbdb91c69 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:06.179299 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1368abc-89ae-4b1f-bae9-6a4bbdb91c69 None admin] RBAC: Adding query filter params (role.id=5d162233989d4e439c441608704dc9b0, scope.project.id=9db0fd07fc494203b2ada13f3cd046d3, user.id=3f20ffd8df994c1db1b1e7248db5b5a3) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:06.179603 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1368abc-89ae-4b1f-bae9-6a4bbdb91c69 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:06.181015 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1368abc-89ae-4b1f-bae9-6a4bbdb91c69 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:06.187151 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 497/982] 10.222.0.52 () {56 vars in 1374 bytes} [Sun Sep 25 00:43:06 2022] GET /identity/v3/role_assignments?user.id=3f20ffd8df994c1db1b1e7248db5b5a3&scope.project.id=9db0fd07fc494203b2ada13f3cd046d3&role.id=5d162233989d4e439c441608704dc9b0 => generated 608 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:06.797765 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d59e79e-3a87-42ed-b406-9b510bb2a532 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:06.797967 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d59e79e-3a87-42ed-b406-9b510bb2a532 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:06.798210 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d59e79e-3a87-42ed-b406-9b510bb2a532 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:06.799676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 486/983] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:43:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:43:06.805751 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744da7c7-eca0-4ee9-91ed-91ff76a7df70 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:06.806028 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744da7c7-eca0-4ee9-91ed-91ff76a7df70 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:06.806292 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744da7c7-eca0-4ee9-91ed-91ff76a7df70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:06.825486 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-744da7c7-eca0-4ee9-91ed-91ff76a7df70 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:43:06.829123 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-744da7c7-eca0-4ee9-91ed-91ff76a7df70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:43:06.911348 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 498/984] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:43:06 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:43:07.033897 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b17ce87-5d33-48d1-b68d-df47317b067f None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:07.034457 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b17ce87-5d33-48d1-b68d-df47317b067f None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:07.034457 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b17ce87-5d33-48d1-b68d-df47317b067f None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:07.035917 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 487/985] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:43:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:43:07.041423 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97928983-1d19-4204-8f3e-93c816dfd47b None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:07.041925 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97928983-1d19-4204-8f3e-93c816dfd47b None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:07.042003 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97928983-1d19-4204-8f3e-93c816dfd47b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:07.061879 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-97928983-1d19-4204-8f3e-93c816dfd47b None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:43:07.062990 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-97928983-1d19-4204-8f3e-93c816dfd47b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:43:07.117427 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:07.136945 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 499/986] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:43:07 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:43:07.186191 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b17ce87-5d33-48d1-b68d-df47317b067f None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:07.274114 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:07.297238 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b17ce87-5d33-48d1-b68d-df47317b067f None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:07.336469 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b1e8ae8-d303-4596-ad25-f2b4c9f03ce7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:07.337016 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1e8ae8-d303-4596-ad25-f2b4c9f03ce7 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:07.337307 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1e8ae8-d303-4596-ad25-f2b4c9f03ce7 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:07.337589 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1e8ae8-d303-4596-ad25-f2b4c9f03ce7 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:07.338160 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b1e8ae8-d303-4596-ad25-f2b4c9f03ce7 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:07.339517 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b1e8ae8-d303-4596-ad25-f2b4c9f03ce7 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:07.348699 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-3b1e8ae8-d303-4596-ad25-f2b4c9f03ce7 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 25 00:43:07.348699 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:07.348699 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 00:43:07.348699 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 00:43:07.348699 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 00:43:07.348699 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 00:43:07.348699 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 00:43:07.348699 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:43:07.348699 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 25 00:43:07.348699 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:43:07.348699 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 00:43:07.348699 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:43:07.348699 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:07.348699 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:43:07.348699 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:43:07.348699 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:43:07.348699 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:43:07.350541 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:43:07.350541 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:43:07.350541 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:43:07.350541 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:43:07.350541 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:43:07.350541 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:43:07.350541 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 00:43:07.350541 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 00:43:07.350541 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 00:43:07.350541 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 00:43:07.350541 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:43:07.350541 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:43:07.350541 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:43:07.350541 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:43:07.350541 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:43:07.350541 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:43:07.352626 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:43:07.352626 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:43:07.352626 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:43:07.352626 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:43:07.352626 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:43:07.352626 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:43:07.352626 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:43:07.352626 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:43:07.352626 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:43:07.352626 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:43:07.352626 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 00:43:07.352626 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:43:07.352626 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 25 00:43:07.352626 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:43:07.355846 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 488/987] 10.222.0.52 () {56 vars in 1126 bytes} [Sun Sep 25 00:43:07 2022] GET /identity/v3/services/object-store => generated 93 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:43:07.356689 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97928983-1d19-4204-8f3e-93c816dfd47b None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:07.406152 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b1e8ae8-d303-4596-ad25-f2b4c9f03ce7 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:07.481497 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97928983-1d19-4204-8f3e-93c816dfd47b None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:07.532008 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc289edf-11ee-41bc-8090-5a1cf971499e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:07.532650 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc289edf-11ee-41bc-8090-5a1cf971499e None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:07.532914 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc289edf-11ee-41bc-8090-5a1cf971499e None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:07.533390 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc289edf-11ee-41bc-8090-5a1cf971499e None admin] PATH_INFO: `/v3/services` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:07.534018 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc289edf-11ee-41bc-8090-5a1cf971499e None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:07.534315 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc289edf-11ee-41bc-8090-5a1cf971499e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:07.535696 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc289edf-11ee-41bc-8090-5a1cf971499e None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:07.548769 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 500/988] 10.222.0.52 () {56 vars in 1122 bytes} [Sun Sep 25 00:43:07 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:07.551333 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc289edf-11ee-41bc-8090-5a1cf971499e None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:07.562280 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b1e8ae8-d303-4596-ad25-f2b4c9f03ce7 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:07.605725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b6254e4-aa23-4e3b-b474-c1bcc6be80cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:07.606354 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6254e4-aa23-4e3b-b474-c1bcc6be80cd None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:07.606662 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6254e4-aa23-4e3b-b474-c1bcc6be80cd None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:07.606955 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6254e4-aa23-4e3b-b474-c1bcc6be80cd None admin] PATH_INFO: `/v3/services/compute` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:07.607565 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b6254e4-aa23-4e3b-b474-c1bcc6be80cd None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:07.608996 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b6254e4-aa23-4e3b-b474-c1bcc6be80cd None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:07.619329 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:07.620350 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-9b6254e4-aa23-4e3b-b474-c1bcc6be80cd None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 25 00:43:07.620350 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:07.620350 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 00:43:07.620350 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 00:43:07.620350 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 00:43:07.620350 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 00:43:07.620350 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 00:43:07.620350 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:43:07.620350 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 25 00:43:07.620350 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:43:07.620350 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 00:43:07.620350 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:43:07.620350 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:07.620350 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:43:07.620350 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:43:07.620350 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:43:07.620350 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:43:07.621524 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:43:07.621524 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:43:07.621524 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:43:07.621524 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:43:07.621524 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:43:07.621524 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:43:07.621524 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 00:43:07.621524 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 00:43:07.621524 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 00:43:07.621524 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 00:43:07.621524 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:43:07.621524 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:43:07.621524 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:43:07.621524 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:43:07.621524 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:43:07.621524 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:43:07.622374 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:43:07.622374 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:43:07.622374 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:43:07.622374 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:43:07.622374 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:43:07.622374 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:43:07.622374 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:43:07.622374 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:43:07.622374 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:43:07.622374 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:43:07.622374 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 00:43:07.622374 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:43:07.622374 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 25 00:43:07.622374 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:43:07.622374 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 489/989] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:43:07 2022] GET /identity/v3/services/compute => generated 88 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:43:07.628284 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b6254e4-aa23-4e3b-b474-c1bcc6be80cd None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:07.642855 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:07.692494 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc289edf-11ee-41bc-8090-5a1cf971499e None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:07.733614 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89710754-aebb-46e3-b67b-8e186c0f68eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:07.734482 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89710754-aebb-46e3-b67b-8e186c0f68eb None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:07.734907 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89710754-aebb-46e3-b67b-8e186c0f68eb None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:07.735255 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89710754-aebb-46e3-b67b-8e186c0f68eb None admin] PATH_INFO: `/v3/services` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:07.735927 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89710754-aebb-46e3-b67b-8e186c0f68eb None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:07.736231 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89710754-aebb-46e3-b67b-8e186c0f68eb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:07.738076 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89710754-aebb-46e3-b67b-8e186c0f68eb None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:07.751185 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 501/990] 10.222.0.52 () {56 vars in 1122 bytes} [Sun Sep 25 00:43:07 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:43:07.752531 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89710754-aebb-46e3-b67b-8e186c0f68eb None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:07.775594 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b6254e4-aa23-4e3b-b474-c1bcc6be80cd None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:07.824883 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd59ff1e-5137-46ec-9e6f-f90142cd0657 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:07.825527 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd59ff1e-5137-46ec-9e6f-f90142cd0657 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:07.825804 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd59ff1e-5137-46ec-9e6f-f90142cd0657 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:07.826128 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd59ff1e-5137-46ec-9e6f-f90142cd0657 None admin] PATH_INFO: `/v3/services` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:07.826705 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd59ff1e-5137-46ec-9e6f-f90142cd0657 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:07.826999 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd59ff1e-5137-46ec-9e6f-f90142cd0657 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:07.828371 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd59ff1e-5137-46ec-9e6f-f90142cd0657 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:07.837291 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 490/991] 10.222.0.52 () {56 vars in 1112 bytes} [Sun Sep 25 00:43:07 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:43:07.843588 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd59ff1e-5137-46ec-9e6f-f90142cd0657 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:07.899019 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89710754-aebb-46e3-b67b-8e186c0f68eb None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:07.919969 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:07.931284 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:07.958030 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214141a6-a2e9-4df1-a8e0-d66d698e6560 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:07.959057 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214141a6-a2e9-4df1-a8e0-d66d698e6560 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:07.959057 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214141a6-a2e9-4df1-a8e0-d66d698e6560 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:07.959290 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214141a6-a2e9-4df1-a8e0-d66d698e6560 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:07.959999 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-214141a6-a2e9-4df1-a8e0-d66d698e6560 None admin] RBAC: Adding query filter params (interface=admin, service_id=23b99b8be13642db92077f78b96e2dc4, region_id=RegionOne) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:07.960297 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-214141a6-a2e9-4df1-a8e0-d66d698e6560 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:07.961762 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-214141a6-a2e9-4df1-a8e0-d66d698e6560 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:07.971683 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 502/992] 10.222.0.52 () {56 vars in 1249 bytes} [Sun Sep 25 00:43:07 2022] GET /identity/v3/endpoints?service_id=23b99b8be13642db92077f78b96e2dc4&interface=admin®ion_id=RegionOne => generated 193 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:43:07.971906 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd59ff1e-5137-46ec-9e6f-f90142cd0657 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:07.972938 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214141a6-a2e9-4df1-a8e0-d66d698e6560 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:08.015407 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848bd83e-6dbd-4e26-a112-a28f52468b1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:08.016018 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848bd83e-6dbd-4e26-a112-a28f52468b1b None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:08.016299 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848bd83e-6dbd-4e26-a112-a28f52468b1b None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:08.016568 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848bd83e-6dbd-4e26-a112-a28f52468b1b None admin] PATH_INFO: `/v3/services` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:08.017232 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-848bd83e-6dbd-4e26-a112-a28f52468b1b None admin] RBAC: Adding query filter params (type=compute) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:08.017566 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-848bd83e-6dbd-4e26-a112-a28f52468b1b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:08.018954 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-848bd83e-6dbd-4e26-a112-a28f52468b1b None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:08.025437 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 491/993] 10.222.0.52 () {56 vars in 1112 bytes} [Sun Sep 25 00:43:07 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:43:08.031336 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848bd83e-6dbd-4e26-a112-a28f52468b1b None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:08.111852 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214141a6-a2e9-4df1-a8e0-d66d698e6560 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:08.158247 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a539741-c75b-4f27-8199-03636a329456 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:08.159146 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a539741-c75b-4f27-8199-03636a329456 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:08.159612 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a539741-c75b-4f27-8199-03636a329456 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:08.160207 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a539741-c75b-4f27-8199-03636a329456 None admin] PATH_INFO: `/v3/services` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:08.161101 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a539741-c75b-4f27-8199-03636a329456 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:08.164611 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a539741-c75b-4f27-8199-03636a329456 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:08.170963 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848bd83e-6dbd-4e26-a112-a28f52468b1b None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:08.172783 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 503/994] 10.222.0.52 () {56 vars in 1087 bytes} [Sun Sep 25 00:43:07 2022] GET /identity/v3/services => generated 1727 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:43:08.236871 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a0b880-d3f2-45c5-b5b6-e3766285dbe6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:08.238064 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a0b880-d3f2-45c5-b5b6-e3766285dbe6 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:08.238492 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a0b880-d3f2-45c5-b5b6-e3766285dbe6 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:08.238906 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a0b880-d3f2-45c5-b5b6-e3766285dbe6 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:08.240188 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a0b880-d3f2-45c5-b5b6-e3766285dbe6 None admin] RBAC: Adding query filter params (interface=public, service_id=50916aacb39d4763aa568327019089ee, region_id=RegionOne) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:08.240694 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a0b880-d3f2-45c5-b5b6-e3766285dbe6 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:08.243135 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a0b880-d3f2-45c5-b5b6-e3766285dbe6 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:08.247924 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:08.259126 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 492/995] 10.222.0.52 () {56 vars in 1251 bytes} [Sun Sep 25 00:43:08 2022] GET /identity/v3/endpoints?service_id=50916aacb39d4763aa568327019089ee&interface=public®ion_id=RegionOne => generated 194 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:43:08.260214 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a539741-c75b-4f27-8199-03636a329456 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:08.310538 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:08.379169 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a539741-c75b-4f27-8199-03636a329456 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:08.422361 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60ca5a60-df04-4144-8f8b-3175307e06ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:08.422880 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ca5a60-df04-4144-8f8b-3175307e06ce None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:08.423155 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ca5a60-df04-4144-8f8b-3175307e06ce None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:08.423419 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ca5a60-df04-4144-8f8b-3175307e06ce None admin] PATH_INFO: `/v3/services` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:08.424021 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60ca5a60-df04-4144-8f8b-3175307e06ce None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:08.425440 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60ca5a60-df04-4144-8f8b-3175307e06ce None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:08.441320 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 504/996] 10.222.0.52 () {56 vars in 1087 bytes} [Sun Sep 25 00:43:08 2022] GET /identity/v3/services => generated 1727 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 00:43:08.603601 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e5a63b5-f5be-45a2-bb47-c8ef686b6f9a None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:08.604111 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e5a63b5-f5be-45a2-bb47-c8ef686b6f9a None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:08.604480 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e5a63b5-f5be-45a2-bb47-c8ef686b6f9a None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:08.606345 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 493/997] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:43:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:43:08.612680 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfd80e6-2ea1-4c19-897b-0319f987e1eb None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:08.613405 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfd80e6-2ea1-4c19-897b-0319f987e1eb None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:08.613807 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfd80e6-2ea1-4c19-897b-0319f987e1eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:08.635842 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-3bfd80e6-2ea1-4c19-897b-0319f987e1eb None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:43:08.637329 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-3bfd80e6-2ea1-4c19-897b-0319f987e1eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:43:08.714452 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 505/998] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:43:08 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 25 00:43:08.996009 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e5a63b5-f5be-45a2-bb47-c8ef686b6f9a None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:09.120940 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e5a63b5-f5be-45a2-bb47-c8ef686b6f9a None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:09.164623 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4965dea-751a-476e-bb47-c2131d96751f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:09.165348 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4965dea-751a-476e-bb47-c2131d96751f None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:09.165675 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4965dea-751a-476e-bb47-c2131d96751f None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:09.165943 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4965dea-751a-476e-bb47-c2131d96751f None admin] PATH_INFO: `/v3/services/image` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:09.166523 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4965dea-751a-476e-bb47-c2131d96751f None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:09.167890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4965dea-751a-476e-bb47-c2131d96751f None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:09.178898 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-a4965dea-751a-476e-bb47-c2131d96751f None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 25 00:43:09.178898 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:09.178898 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 00:43:09.178898 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 00:43:09.178898 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 00:43:09.178898 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 00:43:09.178898 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 00:43:09.178898 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:43:09.178898 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 25 00:43:09.178898 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:43:09.178898 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 00:43:09.178898 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:43:09.178898 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:09.178898 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:43:09.178898 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:43:09.178898 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:43:09.178898 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:43:09.181923 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:43:09.181923 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:43:09.181923 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:43:09.181923 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:43:09.181923 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:43:09.181923 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:43:09.181923 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 00:43:09.181923 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 00:43:09.181923 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 00:43:09.181923 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 00:43:09.181923 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:43:09.181923 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:43:09.181923 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:43:09.181923 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:43:09.181923 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:43:09.181923 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:43:09.183576 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:43:09.183576 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:43:09.183576 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:43:09.183576 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:43:09.183576 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:43:09.183576 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:43:09.183576 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:43:09.183576 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:43:09.183576 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:43:09.183576 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:43:09.183576 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 00:43:09.183576 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:43:09.183576 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 25 00:43:09.183576 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:43:09.187970 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 494/999] 10.222.0.52 () {56 vars in 1105 bytes} [Sun Sep 25 00:43:08 2022] GET /identity/v3/services/image => generated 86 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:43:09.189540 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bfd80e6-2ea1-4c19-897b-0319f987e1eb None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:09.231987 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:09.317077 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bfd80e6-2ea1-4c19-897b-0319f987e1eb None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:09.358673 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e111cc-e4fd-4484-855a-1c22d8df0a18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:09.359268 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e111cc-e4fd-4484-855a-1c22d8df0a18 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:09.359534 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e111cc-e4fd-4484-855a-1c22d8df0a18 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:09.359799 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e111cc-e4fd-4484-855a-1c22d8df0a18 None admin] PATH_INFO: `/v3/services` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:09.360423 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e111cc-e4fd-4484-855a-1c22d8df0a18 None admin] RBAC: Adding query filter params (name=image) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:09.360719 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e111cc-e4fd-4484-855a-1c22d8df0a18 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:09.362104 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e111cc-e4fd-4484-855a-1c22d8df0a18 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:09.369628 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 506/1000] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:43:09 2022] GET /identity/v3/services?name=image => generated 122 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:43:09.375930 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4965dea-751a-476e-bb47-c2131d96751f None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:09.457039 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:09.519435 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4965dea-751a-476e-bb47-c2131d96751f None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:09.564159 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f21e555-e5e1-43a2-8228-2599db5d6e22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:09.564766 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f21e555-e5e1-43a2-8228-2599db5d6e22 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:09.565063 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f21e555-e5e1-43a2-8228-2599db5d6e22 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:09.565543 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f21e555-e5e1-43a2-8228-2599db5d6e22 None admin] PATH_INFO: `/v3/services` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:09.566333 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f21e555-e5e1-43a2-8228-2599db5d6e22 None admin] RBAC: Adding query filter params (type=image) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:09.566806 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f21e555-e5e1-43a2-8228-2599db5d6e22 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:09.569681 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f21e555-e5e1-43a2-8228-2599db5d6e22 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:09.575199 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 495/1001] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:43:09 2022] GET /identity/v3/services?type=image => generated 122 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:43:10.453167 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ed3064-e4f9-4aa1-8c99-09e323ea9ec1 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:10.453761 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ed3064-e4f9-4aa1-8c99-09e323ea9ec1 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:10.453761 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ed3064-e4f9-4aa1-8c99-09e323ea9ec1 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:10.455379 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 507/1002] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:43:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 00:43:10.461236 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0adc764-d25f-4486-a66d-86cf0e2428ff None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:10.461951 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0adc764-d25f-4486-a66d-86cf0e2428ff None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:10.461951 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0adc764-d25f-4486-a66d-86cf0e2428ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:10.488142 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-b0adc764-d25f-4486-a66d-86cf0e2428ff None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:43:10.490219 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-b0adc764-d25f-4486-a66d-86cf0e2428ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:43:10.589292 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 496/1003] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:43:10 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:43:10.780709 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b58dd991-e1d8-449c-af15-55fc1d3be580 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:10.781047 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b58dd991-e1d8-449c-af15-55fc1d3be580 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:10.781286 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b58dd991-e1d8-449c-af15-55fc1d3be580 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:10.782757 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 508/1004] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:43:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 00:43:10.788021 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9d93fd-9060-4699-9b20-f428fcc28154 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:10.788309 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9d93fd-9060-4699-9b20-f428fcc28154 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:10.788581 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9d93fd-9060-4699-9b20-f428fcc28154 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:10.807333 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-3e9d93fd-9060-4699-9b20-f428fcc28154 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:43:10.808510 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-3e9d93fd-9060-4699-9b20-f428fcc28154 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:43:10.866602 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:10.871064 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b58dd991-e1d8-449c-af15-55fc1d3be580 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:10.881609 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 497/1005] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:43:10 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:43:10.978416 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:10.984386 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b58dd991-e1d8-449c-af15-55fc1d3be580 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:11.021372 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea7261ea-75c2-475f-9e19-c635e1cae87a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:11.022057 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7261ea-75c2-475f-9e19-c635e1cae87a None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:11.022280 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7261ea-75c2-475f-9e19-c635e1cae87a None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:11.022570 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7261ea-75c2-475f-9e19-c635e1cae87a None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:11.023117 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea7261ea-75c2-475f-9e19-c635e1cae87a None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:11.024466 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea7261ea-75c2-475f-9e19-c635e1cae87a None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:11.041731 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-ea7261ea-75c2-475f-9e19-c635e1cae87a None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 25 00:43:11.041731 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:11.041731 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 00:43:11.041731 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 00:43:11.041731 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 00:43:11.041731 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 00:43:11.041731 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 00:43:11.041731 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:43:11.041731 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 25 00:43:11.041731 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:43:11.041731 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 00:43:11.041731 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:43:11.041731 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:11.041731 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:43:11.041731 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:43:11.041731 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:43:11.041731 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:43:11.042908 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:43:11.042908 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:43:11.042908 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:43:11.042908 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:43:11.042908 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:43:11.042908 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:43:11.042908 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 00:43:11.042908 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 00:43:11.042908 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 00:43:11.042908 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 00:43:11.042908 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:43:11.042908 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:43:11.042908 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:43:11.042908 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:43:11.042908 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:43:11.042908 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:43:11.044428 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:43:11.044428 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:43:11.044428 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:43:11.044428 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:43:11.044428 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:43:11.044428 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:43:11.044428 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:43:11.044428 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:43:11.044428 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:43:11.044428 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:43:11.044428 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 00:43:11.044428 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:43:11.044428 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 25 00:43:11.044428 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:43:11.049633 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 509/1006] 10.222.0.52 () {56 vars in 1126 bytes} [Sun Sep 25 00:43:10 2022] GET /identity/v3/services/object-store => generated 93 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:43:11.051251 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9d93fd-9060-4699-9b20-f428fcc28154 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:11.152188 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea7261ea-75c2-475f-9e19-c635e1cae87a None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:11.195841 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9d93fd-9060-4699-9b20-f428fcc28154 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:11.238970 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ddc001-9183-4f46-bd15-d768ed86af73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:11.239715 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ddc001-9183-4f46-bd15-d768ed86af73 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:11.239993 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ddc001-9183-4f46-bd15-d768ed86af73 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:11.240716 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ddc001-9183-4f46-bd15-d768ed86af73 None admin] PATH_INFO: `/v3/services` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:11.241389 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0ddc001-9183-4f46-bd15-d768ed86af73 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:11.241696 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0ddc001-9183-4f46-bd15-d768ed86af73 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:11.243424 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0ddc001-9183-4f46-bd15-d768ed86af73 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:11.249295 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 498/1007] 10.222.0.52 () {56 vars in 1122 bytes} [Sun Sep 25 00:43:11 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:11.254604 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ddc001-9183-4f46-bd15-d768ed86af73 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:11.272339 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea7261ea-75c2-475f-9e19-c635e1cae87a None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:11.307529 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecc370cd-ecb2-4bcc-a5aa-3d84b04dfa11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:11.308069 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc370cd-ecb2-4bcc-a5aa-3d84b04dfa11 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:11.308348 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc370cd-ecb2-4bcc-a5aa-3d84b04dfa11 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:11.308612 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc370cd-ecb2-4bcc-a5aa-3d84b04dfa11 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:11.309193 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecc370cd-ecb2-4bcc-a5aa-3d84b04dfa11 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:11.310590 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecc370cd-ecb2-4bcc-a5aa-3d84b04dfa11 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:11.319732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-ecc370cd-ecb2-4bcc-a5aa-3d84b04dfa11 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 25 00:43:11.319732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:11.319732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 00:43:11.319732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 00:43:11.319732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 00:43:11.319732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 00:43:11.319732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 00:43:11.319732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:43:11.319732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 25 00:43:11.319732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:43:11.319732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 00:43:11.319732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:43:11.319732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:11.319732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:43:11.319732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:43:11.319732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:43:11.319732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:43:11.320767 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:43:11.320767 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:43:11.320767 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:43:11.320767 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:43:11.320767 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:43:11.320767 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:43:11.320767 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 00:43:11.320767 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 00:43:11.320767 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 00:43:11.320767 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 00:43:11.320767 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:43:11.320767 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:43:11.320767 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:43:11.320767 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:43:11.320767 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:43:11.320767 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:43:11.322142 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:43:11.322142 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:43:11.322142 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:43:11.322142 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:43:11.322142 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:43:11.322142 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:43:11.322142 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:43:11.322142 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:43:11.322142 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:43:11.322142 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:43:11.322142 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 00:43:11.322142 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:43:11.322142 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 25 00:43:11.322142 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:43:11.323264 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:11.325819 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 510/1008] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:43:11 2022] GET /identity/v3/services/compute => generated 88 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:43:11.326766 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecc370cd-ecb2-4bcc-a5aa-3d84b04dfa11 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:11.356644 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:11.379307 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ddc001-9183-4f46-bd15-d768ed86af73 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:11.416558 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e21cc1d-eb21-4367-a2b2-f91ff7d363e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:11.417320 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e21cc1d-eb21-4367-a2b2-f91ff7d363e3 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:11.417667 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e21cc1d-eb21-4367-a2b2-f91ff7d363e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:11.418012 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e21cc1d-eb21-4367-a2b2-f91ff7d363e3 None admin] PATH_INFO: `/v3/services` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:11.418767 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e21cc1d-eb21-4367-a2b2-f91ff7d363e3 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:11.419156 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e21cc1d-eb21-4367-a2b2-f91ff7d363e3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:11.420601 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e21cc1d-eb21-4367-a2b2-f91ff7d363e3 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:11.426109 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 499/1009] 10.222.0.52 () {56 vars in 1122 bytes} [Sun Sep 25 00:43:11 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:43:11.432282 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e21cc1d-eb21-4367-a2b2-f91ff7d363e3 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:11.449761 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecc370cd-ecb2-4bcc-a5aa-3d84b04dfa11 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:11.500013 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f4dd42-af90-4ab2-9167-9b82f9dfe558 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:11.500974 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f4dd42-af90-4ab2-9167-9b82f9dfe558 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:11.501451 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f4dd42-af90-4ab2-9167-9b82f9dfe558 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:11.501891 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f4dd42-af90-4ab2-9167-9b82f9dfe558 None admin] PATH_INFO: `/v3/services` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:11.503001 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8f4dd42-af90-4ab2-9167-9b82f9dfe558 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:11.503481 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8f4dd42-af90-4ab2-9167-9b82f9dfe558 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:11.506553 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8f4dd42-af90-4ab2-9167-9b82f9dfe558 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:11.519936 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 511/1010] 10.222.0.52 () {56 vars in 1112 bytes} [Sun Sep 25 00:43:11 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:11.521968 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f4dd42-af90-4ab2-9167-9b82f9dfe558 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:11.592447 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e21cc1d-eb21-4367-a2b2-f91ff7d363e3 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:11.610649 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:11.626906 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:11.632822 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06488bdb-eddd-403f-bc3a-8ff39b0b7f34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:11.633527 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06488bdb-eddd-403f-bc3a-8ff39b0b7f34 None admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:11.633756 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06488bdb-eddd-403f-bc3a-8ff39b0b7f34 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:11.634028 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06488bdb-eddd-403f-bc3a-8ff39b0b7f34 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:11.634702 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06488bdb-eddd-403f-bc3a-8ff39b0b7f34 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:11.636072 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06488bdb-eddd-403f-bc3a-8ff39b0b7f34 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:11.638367 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f4dd42-af90-4ab2-9167-9b82f9dfe558 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:11.653645 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 500/1011] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:43:11 2022] POST /identity/v3/endpoints => generated 325 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 00:43:11.685650 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c16211eb-c918-4b69-a23f-171cf010b381 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:11.686274 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16211eb-c918-4b69-a23f-171cf010b381 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:11.686556 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16211eb-c918-4b69-a23f-171cf010b381 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:11.686810 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16211eb-c918-4b69-a23f-171cf010b381 None admin] PATH_INFO: `/v3/services` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:11.687451 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c16211eb-c918-4b69-a23f-171cf010b381 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:11.687749 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c16211eb-c918-4b69-a23f-171cf010b381 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:11.689148 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c16211eb-c918-4b69-a23f-171cf010b381 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:11.700795 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 512/1012] 10.222.0.52 () {56 vars in 1112 bytes} [Sun Sep 25 00:43:11 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:11.702620 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06488bdb-eddd-403f-bc3a-8ff39b0b7f34 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:11.841843 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0307fc46-1702-42a0-8241-6c1e4afbbaa6 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:11.842122 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0307fc46-1702-42a0-8241-6c1e4afbbaa6 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:11.842383 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0307fc46-1702-42a0-8241-6c1e4afbbaa6 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:11.848070 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 513/1013] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:43:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 00:43:11.850272 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd1ac27-44f2-4351-aad3-b4bc79e1837c None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:11.850552 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd1ac27-44f2-4351-aad3-b4bc79e1837c None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:11.850822 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd1ac27-44f2-4351-aad3-b4bc79e1837c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:11.857829 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06488bdb-eddd-403f-bc3a-8ff39b0b7f34 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:11.874679 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-8cd1ac27-44f2-4351-aad3-b4bc79e1837c None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:43:11.875891 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-8cd1ac27-44f2-4351-aad3-b4bc79e1837c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:43:11.904293 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ebf7629-672d-4c10-a71e-e2f13daa7f44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:11.905015 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ebf7629-672d-4c10-a71e-e2f13daa7f44 None admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:11.905320 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ebf7629-672d-4c10-a71e-e2f13daa7f44 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:11.905624 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ebf7629-672d-4c10-a71e-e2f13daa7f44 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:11.906365 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ebf7629-672d-4c10-a71e-e2f13daa7f44 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:11.907731 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ebf7629-672d-4c10-a71e-e2f13daa7f44 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:11.937293 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 501/1014] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:43:11 2022] POST /identity/v3/endpoints => generated 334 bytes in 229 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:43:11.964121 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 514/1015] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:43:11 2022] POST /identity/v3/auth/tokens => generated 1873 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:43:12.239079 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ebf7629-672d-4c10-a71e-e2f13daa7f44 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:12.275026 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:12.352427 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ebf7629-672d-4c10-a71e-e2f13daa7f44 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:12.393552 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7306fc0e-f9b1-4d19-8cf9-8522242e8d3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:12.394167 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7306fc0e-f9b1-4d19-8cf9-8522242e8d3b None admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:12.394438 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7306fc0e-f9b1-4d19-8cf9-8522242e8d3b None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:12.394701 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7306fc0e-f9b1-4d19-8cf9-8522242e8d3b None admin] PATH_INFO: `/v3/services` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:12.395355 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7306fc0e-f9b1-4d19-8cf9-8522242e8d3b None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:12.397693 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7306fc0e-f9b1-4d19-8cf9-8522242e8d3b None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:12.415893 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 502/1016] 10.222.0.52 () {60 vars in 1141 bytes} [Sun Sep 25 00:43:12 2022] POST /identity/v3/services => generated 242 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:43:13.918661 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e795eb-0f76-4cac-ad9d-cf1a3a19f7f4 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:13.919158 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e795eb-0f76-4cac-ad9d-cf1a3a19f7f4 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:13.919158 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e795eb-0f76-4cac-ad9d-cf1a3a19f7f4 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:13.920438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 515/1017] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:43:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:43:13.926047 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568a7961-c173-4e0f-94de-b97e15d75ff0 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:13.926498 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568a7961-c173-4e0f-94de-b97e15d75ff0 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:13.926579 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568a7961-c173-4e0f-94de-b97e15d75ff0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:13.945650 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-568a7961-c173-4e0f-94de-b97e15d75ff0 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:43:13.946847 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-568a7961-c173-4e0f-94de-b97e15d75ff0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:43:14.023405 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 503/1018] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:43:13 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:43:14.243039 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d1164d-183b-49de-9d72-9f84d7aee4fc None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:14.243438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d1164d-183b-49de-9d72-9f84d7aee4fc None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:14.243616 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d1164d-183b-49de-9d72-9f84d7aee4fc None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:14.245030 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 516/1019] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:43:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:43:14.250359 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb4999a-1cd3-4c74-9b35-ca105fa2f0c1 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:14.250644 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb4999a-1cd3-4c74-9b35-ca105fa2f0c1 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:14.250911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb4999a-1cd3-4c74-9b35-ca105fa2f0c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:14.270008 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-9bb4999a-1cd3-4c74-9b35-ca105fa2f0c1 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:43:14.271350 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-9bb4999a-1cd3-4c74-9b35-ca105fa2f0c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:43:14.285183 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:14.294510 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7d1164d-183b-49de-9d72-9f84d7aee4fc None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:14.345916 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 504/1020] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:43:14 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:43:14.382757 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:14.408849 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7d1164d-183b-49de-9d72-9f84d7aee4fc None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:14.446363 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd731d7-06d8-4303-88dd-2d7bf2281fef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:14.447001 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd731d7-06d8-4303-88dd-2d7bf2281fef None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:14.447274 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd731d7-06d8-4303-88dd-2d7bf2281fef None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:14.447541 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd731d7-06d8-4303-88dd-2d7bf2281fef None admin] PATH_INFO: `/v3/domains/default` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:14.448920 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fd731d7-06d8-4303-88dd-2d7bf2281fef None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:14.450432 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fd731d7-06d8-4303-88dd-2d7bf2281fef None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:14.453284 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 517/1021] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:43:14 2022] GET /identity/v3/domains/default => generated 199 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:14.458494 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb4999a-1cd3-4c74-9b35-ca105fa2f0c1 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:14.578235 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb4999a-1cd3-4c74-9b35-ca105fa2f0c1 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:14.613900 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a429fb-5f62-49a5-9cd4-8009250cfc8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:14.615135 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a429fb-5f62-49a5-9cd4-8009250cfc8d None admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:14.615409 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a429fb-5f62-49a5-9cd4-8009250cfc8d None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:14.615658 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a429fb-5f62-49a5-9cd4-8009250cfc8d None admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:14.616307 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a429fb-5f62-49a5-9cd4-8009250cfc8d None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:14.617766 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a429fb-5f62-49a5-9cd4-8009250cfc8d None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:14.620040 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd731d7-06d8-4303-88dd-2d7bf2281fef None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:14.643232 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:14.651474 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 505/1022] 10.222.0.52 () {60 vars in 1141 bytes} [Sun Sep 25 00:43:14 2022] POST /identity/v3/projects => generated 311 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 00:43:14.652722 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b2c47b-55c8-4335-bba2-601940cb7bfd None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:14.652998 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b2c47b-55c8-4335-bba2-601940cb7bfd None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:14.653281 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b2c47b-55c8-4335-bba2-601940cb7bfd None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:14.654499 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 506/1023] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:43:14 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:43:14.664312 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0437cabb-46f2-460a-bd10-72a06dfe1a10 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:14.664636 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0437cabb-46f2-460a-bd10-72a06dfe1a10 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:14.664858 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0437cabb-46f2-460a-bd10-72a06dfe1a10 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:14.685694 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-0437cabb-46f2-460a-bd10-72a06dfe1a10 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:43:14.686925 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-0437cabb-46f2-460a-bd10-72a06dfe1a10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:43:14.746219 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd731d7-06d8-4303-88dd-2d7bf2281fef None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:14.769303 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 507/1024] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:43:14 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:43:14.789447 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de920934-e15d-4a98-9b5b-63fac592b4ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:14.790099 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de920934-e15d-4a98-9b5b-63fac592b4ab None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:14.790376 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de920934-e15d-4a98-9b5b-63fac592b4ab None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:14.790644 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de920934-e15d-4a98-9b5b-63fac592b4ab None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:14.802919 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de920934-e15d-4a98-9b5b-63fac592b4ab None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:43:14.807960 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-de920934-e15d-4a98-9b5b-63fac592b4ab None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 25 00:43:14.807960 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:14.807960 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:43:14.807960 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:43:14.807960 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:43:14.807960 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:43:14.807960 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:43:14.807960 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:43:14.807960 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:43:14.807960 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:43:14.807960 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:43:14.807960 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:43:14.807960 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:43:14.807960 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:43:14.807960 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:43:14.807960 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:43:14.808979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:43:14.808979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:43:14.808979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:43:14.808979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:43:14.808979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:43:14.808979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:43:14.808979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:43:14.808979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:43:14.808979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:43:14.808979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:43:14.808979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:43:14.808979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:43:14.808979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:43:14.808979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:43:14.808979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:43:14.808979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:43:14.810091 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:43:14.810091 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:43:14.810091 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:43:14.810091 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:43:14.810091 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:43:14.810091 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:43:14.810091 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:43:14.810091 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:43:14.810091 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:43:14.810091 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:43:14.810091 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 25 00:43:14.810091 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:43:14.810091 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 518/1025] 10.222.0.52 () {56 vars in 1120 bytes} [Sun Sep 25 00:43:14 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:43:14.819044 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0437cabb-46f2-460a-bd10-72a06dfe1a10 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:14.931795 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0437cabb-46f2-460a-bd10-72a06dfe1a10 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:14.963543 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:14.970787 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3256c349-822c-4708-84f3-aa4b50626aa2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:14.971728 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3256c349-822c-4708-84f3-aa4b50626aa2 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:14.972218 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3256c349-822c-4708-84f3-aa4b50626aa2 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:14.972733 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3256c349-822c-4708-84f3-aa4b50626aa2 None admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:14.974177 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3256c349-822c-4708-84f3-aa4b50626aa2 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:14.974704 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3256c349-822c-4708-84f3-aa4b50626aa2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:14.976290 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3256c349-822c-4708-84f3-aa4b50626aa2 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:14.985666 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 508/1026] 10.222.0.52 () {56 vars in 1115 bytes} [Sun Sep 25 00:43:14 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:14.992183 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de920934-e15d-4a98-9b5b-63fac592b4ab None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:14.997474 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:15.052927 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3256c349-822c-4708-84f3-aa4b50626aa2 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:15.104350 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de920934-e15d-4a98-9b5b-63fac592b4ab None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:15.140925 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c3d79f-a033-4769-b50b-8ff5ef794a76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:15.141575 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c3d79f-a033-4769-b50b-8ff5ef794a76 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:15.141844 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c3d79f-a033-4769-b50b-8ff5ef794a76 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:15.142105 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c3d79f-a033-4769-b50b-8ff5ef794a76 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:15.153801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18c3d79f-a033-4769-b50b-8ff5ef794a76 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:15.155361 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18c3d79f-a033-4769-b50b-8ff5ef794a76 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:15.171217 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-18c3d79f-a033-4769-b50b-8ff5ef794a76 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:43:15.171217 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:15.171217 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:43:15.171217 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:43:15.171217 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:43:15.171217 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:43:15.171217 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:43:15.171217 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:43:15.171217 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:43:15.171217 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:43:15.171217 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:43:15.171217 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:43:15.171217 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:43:15.171217 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:43:15.171217 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:43:15.171217 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:43:15.172390 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:43:15.172390 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:43:15.172390 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:43:15.172390 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:43:15.172390 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:43:15.172390 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:43:15.172390 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:43:15.172390 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:43:15.172390 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:43:15.172390 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:43:15.172390 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:43:15.172390 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:43:15.172390 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:43:15.172390 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:43:15.172390 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:43:15.172390 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:43:15.172390 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:43:15.173442 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:43:15.173442 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:43:15.173442 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:43:15.173632 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3256c349-822c-4708-84f3-aa4b50626aa2 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:15.177526 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 519/1027] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:43:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:43:15.178707 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c3d79f-a033-4769-b50b-8ff5ef794a76 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:15.211022 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4ea0b2-5ee2-47fd-b9f6-5da62bfbe005 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:15.211571 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4ea0b2-5ee2-47fd-b9f6-5da62bfbe005 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:15.211840 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4ea0b2-5ee2-47fd-b9f6-5da62bfbe005 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:15.212104 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4ea0b2-5ee2-47fd-b9f6-5da62bfbe005 None admin] PATH_INFO: `/v3/services/image` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:15.212683 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b4ea0b2-5ee2-47fd-b9f6-5da62bfbe005 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:15.214065 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b4ea0b2-5ee2-47fd-b9f6-5da62bfbe005 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:15.223096 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-3b4ea0b2-5ee2-47fd-b9f6-5da62bfbe005 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 25 00:43:15.223096 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:15.223096 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 00:43:15.223096 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 00:43:15.223096 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 00:43:15.223096 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 00:43:15.223096 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 00:43:15.223096 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:43:15.223096 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 25 00:43:15.223096 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:43:15.223096 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 00:43:15.223096 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:43:15.223096 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:15.223096 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:43:15.223096 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:43:15.223096 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:43:15.223096 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:43:15.224911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:43:15.224911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:43:15.224911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:43:15.224911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:43:15.224911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:43:15.224911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:43:15.224911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 00:43:15.224911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 00:43:15.224911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 00:43:15.224911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 00:43:15.224911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:43:15.224911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:43:15.224911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:43:15.224911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:43:15.224911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:43:15.224911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:43:15.230864 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:43:15.230864 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:43:15.230864 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:43:15.230864 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:43:15.230864 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:43:15.230864 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:43:15.230864 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:43:15.230864 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:43:15.230864 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:43:15.230864 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:43:15.230864 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 00:43:15.230864 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:43:15.230864 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 25 00:43:15.230864 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:43:15.230864 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 509/1028] 10.222.0.52 () {56 vars in 1105 bytes} [Sun Sep 25 00:43:15 2022] GET /identity/v3/services/image => generated 86 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:43:15.231693 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4ea0b2-5ee2-47fd-b9f6-5da62bfbe005 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:15.248798 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:15.291332 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:15.300397 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c3d79f-a033-4769-b50b-8ff5ef794a76 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:15.336652 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d5d3fe-d7ef-4632-adb8-79079735e846 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:15.337246 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d5d3fe-d7ef-4632-adb8-79079735e846 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:15.337542 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d5d3fe-d7ef-4632-adb8-79079735e846 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:15.337803 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d5d3fe-d7ef-4632-adb8-79079735e846 None admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:15.338426 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92d5d3fe-d7ef-4632-adb8-79079735e846 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:15.338720 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92d5d3fe-d7ef-4632-adb8-79079735e846 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:15.338951 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4ea0b2-5ee2-47fd-b9f6-5da62bfbe005 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:15.340055 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92d5d3fe-d7ef-4632-adb8-79079735e846 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:15.353324 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 520/1029] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:43:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:15.358631 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d5d3fe-d7ef-4632-adb8-79079735e846 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:15.379203 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6226665-3ec2-4636-a769-125921a9d579 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:15.380453 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6226665-3ec2-4636-a769-125921a9d579 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:15.380735 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6226665-3ec2-4636-a769-125921a9d579 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:15.381005 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6226665-3ec2-4636-a769-125921a9d579 None admin] PATH_INFO: `/v3/services` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:15.381678 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6226665-3ec2-4636-a769-125921a9d579 None admin] RBAC: Adding query filter params (name=image) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:15.382049 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6226665-3ec2-4636-a769-125921a9d579 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:15.383429 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6226665-3ec2-4636-a769-125921a9d579 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:15.388603 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 510/1030] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:43:15 2022] GET /identity/v3/services?name=image => generated 122 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:43:15.394185 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6226665-3ec2-4636-a769-125921a9d579 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:15.485605 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d5d3fe-d7ef-4632-adb8-79079735e846 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:15.507698 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:15.511631 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6226665-3ec2-4636-a769-125921a9d579 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:15.529379 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c9541b-e192-4564-91cf-d081bbfec525 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:15.530219 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c9541b-e192-4564-91cf-d081bbfec525 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:15.530489 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c9541b-e192-4564-91cf-d081bbfec525 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:15.530752 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c9541b-e192-4564-91cf-d081bbfec525 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:15.537850 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:15.539839 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09c9541b-e192-4564-91cf-d081bbfec525 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:15.541319 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09c9541b-e192-4564-91cf-d081bbfec525 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:15.555737 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-09c9541b-e192-4564-91cf-d081bbfec525 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 25 00:43:15.555737 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:15.555737 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:43:15.555737 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:43:15.555737 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:43:15.555737 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:43:15.555737 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:43:15.555737 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:43:15.555737 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:43:15.555737 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:43:15.555737 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:43:15.555737 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:43:15.555737 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 00:43:15.555737 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 00:43:15.555737 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 00:43:15.555737 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 00:43:15.556949 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:43:15.556949 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:43:15.556949 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 00:43:15.556949 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:43:15.556949 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 00:43:15.556949 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:43:15.556949 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:43:15.556949 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:43:15.556949 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:43:15.556949 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:43:15.556949 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:43:15.556949 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:43:15.556949 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:43:15.556949 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:43:15.556949 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:43:15.556949 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:43:15.556949 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:43:15.556949 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:43:15.558296 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:43:15.558296 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:43:15.558296 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 00:43:15.558296 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 00:43:15.558296 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 00:43:15.558296 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 00:43:15.558296 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 00:43:15.558296 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 00:43:15.558296 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 25 00:43:15.558296 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:43:15.558988 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ddc2b11-2fff-49ca-b02d-8ab5d1904d68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:15.558988 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ddc2b11-2fff-49ca-b02d-8ab5d1904d68 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:15.558988 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ddc2b11-2fff-49ca-b02d-8ab5d1904d68 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:15.559456 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ddc2b11-2fff-49ca-b02d-8ab5d1904d68 None admin] PATH_INFO: `/v3/services` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:15.560128 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ddc2b11-2fff-49ca-b02d-8ab5d1904d68 None admin] RBAC: Adding query filter params (type=image) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:15.560500 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ddc2b11-2fff-49ca-b02d-8ab5d1904d68 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:15.561772 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 521/1031] 10.222.0.52 () {56 vars in 1093 bytes} [Sun Sep 25 00:43:15 2022] GET /identity/v3/users/nova => generated 82 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:43:15.562052 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ddc2b11-2fff-49ca-b02d-8ab5d1904d68 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:15.562428 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c9541b-e192-4564-91cf-d081bbfec525 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:15.573627 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 511/1032] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:43:15 2022] GET /identity/v3/services?type=image => generated 350 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:43:15.575261 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ddc2b11-2fff-49ca-b02d-8ab5d1904d68 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:15.676867 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c9541b-e192-4564-91cf-d081bbfec525 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:15.697517 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ddc2b11-2fff-49ca-b02d-8ab5d1904d68 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:15.716165 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1928fa5f-4cc7-423a-8f2c-52d71a1aa7d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:15.716766 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1928fa5f-4cc7-423a-8f2c-52d71a1aa7d7 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:15.717037 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1928fa5f-4cc7-423a-8f2c-52d71a1aa7d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:15.717324 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1928fa5f-4cc7-423a-8f2c-52d71a1aa7d7 None admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:15.718061 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1928fa5f-4cc7-423a-8f2c-52d71a1aa7d7 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:15.718361 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1928fa5f-4cc7-423a-8f2c-52d71a1aa7d7 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:15.719828 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1928fa5f-4cc7-423a-8f2c-52d71a1aa7d7 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:15.737338 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db613a3f-64d9-44d5-8b31-c63f5475f8f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:15.738144 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db613a3f-64d9-44d5-8b31-c63f5475f8f0 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:15.738420 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db613a3f-64d9-44d5-8b31-c63f5475f8f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:15.738690 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db613a3f-64d9-44d5-8b31-c63f5475f8f0 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:15.739938 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db613a3f-64d9-44d5-8b31-c63f5475f8f0 None admin] RBAC: Adding query filter params (interface=public, service_id=c36f9d3cc9cd45b0a3db675be6b5b979, region_id=RegionOne) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:15.741064 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db613a3f-64d9-44d5-8b31-c63f5475f8f0 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:15.742678 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db613a3f-64d9-44d5-8b31-c63f5475f8f0 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:15.744763 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 522/1033] 10.222.0.52 () {56 vars in 1133 bytes} [Sun Sep 25 00:43:15 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:43:15.750287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 512/1034] 10.222.0.52 () {56 vars in 1251 bytes} [Sun Sep 25 00:43:15 2022] GET /identity/v3/endpoints?service_id=c36f9d3cc9cd45b0a3db675be6b5b979&interface=public®ion_id=RegionOne => generated 194 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:15.750628 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1928fa5f-4cc7-423a-8f2c-52d71a1aa7d7 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:15.756974 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db613a3f-64d9-44d5-8b31-c63f5475f8f0 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:15.792460 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:15.825817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:15.886156 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1928fa5f-4cc7-423a-8f2c-52d71a1aa7d7 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:15.888299 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db613a3f-64d9-44d5-8b31-c63f5475f8f0 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:15.929884 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a886497-93c2-464f-9ee7-ec9ccce1e8a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:15.930204 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce27c727-df39-4d0b-b43b-7afde052393c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:15.930658 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a886497-93c2-464f-9ee7-ec9ccce1e8a1 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:15.930821 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce27c727-df39-4d0b-b43b-7afde052393c None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:15.931109 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a886497-93c2-464f-9ee7-ec9ccce1e8a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:15.931252 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce27c727-df39-4d0b-b43b-7afde052393c None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:15.931526 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce27c727-df39-4d0b-b43b-7afde052393c None admin] PATH_INFO: `/v3/services` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:15.931625 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a886497-93c2-464f-9ee7-ec9ccce1e8a1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:15.932188 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce27c727-df39-4d0b-b43b-7afde052393c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:15.933528 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce27c727-df39-4d0b-b43b-7afde052393c None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:15.953744 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 513/1035] 10.222.0.52 () {56 vars in 1087 bytes} [Sun Sep 25 00:43:15 2022] GET /identity/v3/services => generated 1957 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 00:43:16.016619 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a886497-93c2-464f-9ee7-ec9ccce1e8a1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:16.018052 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a886497-93c2-464f-9ee7-ec9ccce1e8a1 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:16.059717 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:16.139991 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 523/1036] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:43:15 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 390 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 25 00:43:16.146493 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce27c727-df39-4d0b-b43b-7afde052393c None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:16.278207 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce27c727-df39-4d0b-b43b-7afde052393c None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:16.296879 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:16.323089 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f95b3d-34b5-4e23-b2c7-952b540dfedc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:16.323717 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f95b3d-34b5-4e23-b2c7-952b540dfedc None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:16.323990 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f95b3d-34b5-4e23-b2c7-952b540dfedc None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:16.324254 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f95b3d-34b5-4e23-b2c7-952b540dfedc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:16.337352 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32f95b3d-34b5-4e23-b2c7-952b540dfedc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:16.338792 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32f95b3d-34b5-4e23-b2c7-952b540dfedc None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:16.357037 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-32f95b3d-34b5-4e23-b2c7-952b540dfedc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:43:16.357037 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:16.357037 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:43:16.357037 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:43:16.357037 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:43:16.357037 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:43:16.357037 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:43:16.357037 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:43:16.357037 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:43:16.357037 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:43:16.357037 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:43:16.357037 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:43:16.357037 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:43:16.357037 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:43:16.357037 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:43:16.357037 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:43:16.358253 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:43:16.358253 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:43:16.358253 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:43:16.358253 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:43:16.358253 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:43:16.358253 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:43:16.358253 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:43:16.358253 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:43:16.358253 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:43:16.358253 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:43:16.358253 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:43:16.358253 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:43:16.358253 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:43:16.358253 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:43:16.358253 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:43:16.358253 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:43:16.358253 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:43:16.359243 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:43:16.359243 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:43:16.359243 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:43:16.359243 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 514/1037] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:43:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:43:16.364868 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a886497-93c2-464f-9ee7-ec9ccce1e8a1 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:16.492484 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a886497-93c2-464f-9ee7-ec9ccce1e8a1 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:16.538273 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75290081-c102-49d9-981f-87ba4e607571 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:16.538853 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75290081-c102-49d9-981f-87ba4e607571 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:16.539130 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75290081-c102-49d9-981f-87ba4e607571 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:16.539395 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75290081-c102-49d9-981f-87ba4e607571 None admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:16.540035 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75290081-c102-49d9-981f-87ba4e607571 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:16.540338 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75290081-c102-49d9-981f-87ba4e607571 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:16.541732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75290081-c102-49d9-981f-87ba4e607571 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:16.543545 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:16.561146 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 524/1038] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:43:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:43:16.562425 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f95b3d-34b5-4e23-b2c7-952b540dfedc None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:16.679677 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f95b3d-34b5-4e23-b2c7-952b540dfedc None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:16.718645 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a8247e-d117-4a95-9259-a33f6374132a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:16.719270 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a8247e-d117-4a95-9259-a33f6374132a None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:16.719540 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a8247e-d117-4a95-9259-a33f6374132a None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:16.719802 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a8247e-d117-4a95-9259-a33f6374132a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:16.729956 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18a8247e-d117-4a95-9259-a33f6374132a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:16.732353 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18a8247e-d117-4a95-9259-a33f6374132a None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:16.749756 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-18a8247e-d117-4a95-9259-a33f6374132a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:43:16.749756 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:16.749756 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:43:16.749756 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:43:16.749756 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:43:16.749756 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:43:16.749756 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:43:16.749756 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:43:16.749756 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:43:16.749756 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:43:16.749756 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:43:16.749756 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:43:16.749756 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 00:43:16.749756 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 00:43:16.749756 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 00:43:16.750760 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 00:43:16.750760 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:43:16.750760 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:43:16.750760 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:43:16.750760 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:43:16.750760 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:43:16.750760 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:43:16.750760 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:43:16.750760 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:43:16.750760 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:43:16.750760 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:43:16.750760 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:43:16.750760 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:43:16.750760 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:43:16.750760 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:43:16.750760 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:43:16.750760 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:43:16.751655 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 00:43:16.751655 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 00:43:16.751655 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 00:43:16.751655 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 00:43:16.751655 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 00:43:16.751655 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 00:43:16.751655 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:43:16.751655 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:43:16.752754 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 515/1039] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:43:16 2022] GET /identity/v3/projects/service => generated 88 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:43:16.757674 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75290081-c102-49d9-981f-87ba4e607571 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:16.863336 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75290081-c102-49d9-981f-87ba4e607571 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:16.899550 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae16251-e2be-434e-a8e6-2d87b21ec77d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:16.900158 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae16251-e2be-434e-a8e6-2d87b21ec77d None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:16.900433 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae16251-e2be-434e-a8e6-2d87b21ec77d None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:16.900699 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae16251-e2be-434e-a8e6-2d87b21ec77d None admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:16.901413 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ae16251-e2be-434e-a8e6-2d87b21ec77d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:16.901718 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ae16251-e2be-434e-a8e6-2d87b21ec77d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:16.903141 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ae16251-e2be-434e-a8e6-2d87b21ec77d None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:16.921928 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 525/1040] 10.222.0.52 () {56 vars in 1148 bytes} [Sun Sep 25 00:43:16 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:43:16.923275 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a8247e-d117-4a95-9259-a33f6374132a None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:16.953803 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:17.051295 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a8247e-d117-4a95-9259-a33f6374132a None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:17.095316 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa0af6e-82f6-44dc-b5fd-04d4c76ee033 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:17.095926 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa0af6e-82f6-44dc-b5fd-04d4c76ee033 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:17.096286 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa0af6e-82f6-44dc-b5fd-04d4c76ee033 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:17.096571 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa0af6e-82f6-44dc-b5fd-04d4c76ee033 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:17.097450 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa0af6e-82f6-44dc-b5fd-04d4c76ee033 None admin] RBAC: Adding query filter params (role.id=5d162233989d4e439c441608704dc9b0, scope.project.id=9db0fd07fc494203b2ada13f3cd046d3, user.id=f142988a4dd14a5f96eacea45c046f85) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:17.097730 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa0af6e-82f6-44dc-b5fd-04d4c76ee033 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:17.100164 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa0af6e-82f6-44dc-b5fd-04d4c76ee033 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:17.113656 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 516/1041] 10.222.0.52 () {56 vars in 1374 bytes} [Sun Sep 25 00:43:16 2022] GET /identity/v3/role_assignments?user.id=f142988a4dd14a5f96eacea45c046f85&scope.project.id=9db0fd07fc494203b2ada13f3cd046d3&role.id=5d162233989d4e439c441608704dc9b0 => generated 259 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:17.115979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1c127f-4e0a-464c-a588-7002a4abbfb5 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:17.116491 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1c127f-4e0a-464c-a588-7002a4abbfb5 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:17.116655 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1c127f-4e0a-464c-a588-7002a4abbfb5 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:17.118198 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 526/1042] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:43:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:43:17.125995 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11499934-6421-4d2c-bd80-dc1a93975dac None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:17.126277 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11499934-6421-4d2c-bd80-dc1a93975dac None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:17.126592 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11499934-6421-4d2c-bd80-dc1a93975dac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:17.152781 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-11499934-6421-4d2c-bd80-dc1a93975dac None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:43:17.154025 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-11499934-6421-4d2c-bd80-dc1a93975dac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:43:17.245545 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 517/1043] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:43:17 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:43:17.524026 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a1c127f-4e0a-464c-a588-7002a4abbfb5 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:17.589070 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:17.646084 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a1c127f-4e0a-464c-a588-7002a4abbfb5 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:17.684982 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c0a962-971e-4b15-81a8-79f4b4754ffd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:17.685801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c0a962-971e-4b15-81a8-79f4b4754ffd None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:17.686113 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c0a962-971e-4b15-81a8-79f4b4754ffd None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:17.686518 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c0a962-971e-4b15-81a8-79f4b4754ffd None admin] PATH_INFO: `/v3/domains/default` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:17.687652 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7c0a962-971e-4b15-81a8-79f4b4754ffd None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:17.689055 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7c0a962-971e-4b15-81a8-79f4b4754ffd None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:17.691218 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 527/1044] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:43:17 2022] GET /identity/v3/domains/default => generated 199 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:17.697613 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11499934-6421-4d2c-bd80-dc1a93975dac None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:17.723700 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:17.825904 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11499934-6421-4d2c-bd80-dc1a93975dac None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:17.865746 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85bd998f-94a7-4812-b069-514d54c750c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:17.866715 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85bd998f-94a7-4812-b069-514d54c750c4 None admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:17.867224 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85bd998f-94a7-4812-b069-514d54c750c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:17.867698 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85bd998f-94a7-4812-b069-514d54c750c4 None admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:17.868630 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85bd998f-94a7-4812-b069-514d54c750c4 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:17.870282 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85bd998f-94a7-4812-b069-514d54c750c4 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:17.895539 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 518/1045] 10.222.0.52 () {60 vars in 1132 bytes} [Sun Sep 25 00:43:17 2022] POST /identity/v3/users => generated 285 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:43:18.319862 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b906e6a-b62d-41ca-9509-bacf24e3f8a9 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:18.320525 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b906e6a-b62d-41ca-9509-bacf24e3f8a9 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:18.320525 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b906e6a-b62d-41ca-9509-bacf24e3f8a9 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:18.322133 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 528/1046] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:43:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:43:18.327857 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fb30b5-7593-4a34-aaa2-a63b32a5b122 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:18.328364 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fb30b5-7593-4a34-aaa2-a63b32a5b122 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:18.328479 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fb30b5-7593-4a34-aaa2-a63b32a5b122 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:18.350061 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-80fb30b5-7593-4a34-aaa2-a63b32a5b122 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:43:18.351480 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-80fb30b5-7593-4a34-aaa2-a63b32a5b122 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:43:18.435709 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 519/1047] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:43:18 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:43:18.731184 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b906e6a-b62d-41ca-9509-bacf24e3f8a9 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:18.841707 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b906e6a-b62d-41ca-9509-bacf24e3f8a9 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:18.867756 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:18.883919 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74a4c6e4-2e34-43e6-8767-ee0b87db4a8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:18.884602 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a4c6e4-2e34-43e6-8767-ee0b87db4a8b None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:18.884870 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a4c6e4-2e34-43e6-8767-ee0b87db4a8b None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:18.885131 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a4c6e4-2e34-43e6-8767-ee0b87db4a8b None admin] PATH_INFO: `/v3/services/image` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:18.885771 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74a4c6e4-2e34-43e6-8767-ee0b87db4a8b None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:18.887183 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74a4c6e4-2e34-43e6-8767-ee0b87db4a8b None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:18.898351 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-74a4c6e4-2e34-43e6-8767-ee0b87db4a8b None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 25 00:43:18.898351 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:18.898351 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 00:43:18.898351 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 00:43:18.898351 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 00:43:18.898351 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 00:43:18.898351 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 00:43:18.898351 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:43:18.898351 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 25 00:43:18.898351 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:43:18.898351 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 00:43:18.898351 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:43:18.898351 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:18.898351 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:43:18.898351 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:43:18.898351 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:43:18.898351 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:43:18.899464 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:43:18.899464 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:43:18.899464 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:43:18.899464 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:43:18.899464 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:43:18.899464 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:43:18.899464 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 00:43:18.899464 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 00:43:18.899464 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 00:43:18.899464 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 00:43:18.899464 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:43:18.899464 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:43:18.899464 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:43:18.899464 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:43:18.899464 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:43:18.899464 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:43:18.900369 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:43:18.900369 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:43:18.900369 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:43:18.900369 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:43:18.900369 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:43:18.900369 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:43:18.900369 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:43:18.900369 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:43:18.900369 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:43:18.900369 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:43:18.900369 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 00:43:18.900369 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:43:18.900369 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 25 00:43:18.900369 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:43:18.904590 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 529/1048] 10.222.0.52 () {56 vars in 1105 bytes} [Sun Sep 25 00:43:18 2022] GET /identity/v3/services/image => generated 86 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:43:18.906002 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80fb30b5-7593-4a34-aaa2-a63b32a5b122 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:18.909192 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:19.027741 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80fb30b5-7593-4a34-aaa2-a63b32a5b122 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:19.067270 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767d6571-1abd-42ee-acf0-49df143366f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:19.068581 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767d6571-1abd-42ee-acf0-49df143366f9 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:19.068878 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767d6571-1abd-42ee-acf0-49df143366f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:19.069154 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767d6571-1abd-42ee-acf0-49df143366f9 None admin] PATH_INFO: `/v3/services` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:19.069858 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-767d6571-1abd-42ee-acf0-49df143366f9 None admin] RBAC: Adding query filter params (name=image) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:19.070159 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-767d6571-1abd-42ee-acf0-49df143366f9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:19.071558 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-767d6571-1abd-42ee-acf0-49df143366f9 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:19.081856 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 520/1049] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:43:18 2022] GET /identity/v3/services?name=image => generated 122 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:19.083095 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74a4c6e4-2e34-43e6-8767-ee0b87db4a8b None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:19.196645 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74a4c6e4-2e34-43e6-8767-ee0b87db4a8b None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:19.237164 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4bcfe00-8445-4135-af1f-cbb7b7e18392 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:19.238004 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4bcfe00-8445-4135-af1f-cbb7b7e18392 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:19.238279 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4bcfe00-8445-4135-af1f-cbb7b7e18392 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:19.238544 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4bcfe00-8445-4135-af1f-cbb7b7e18392 None admin] PATH_INFO: `/v3/services` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:19.239773 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4bcfe00-8445-4135-af1f-cbb7b7e18392 None admin] RBAC: Adding query filter params (type=image) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:19.240075 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4bcfe00-8445-4135-af1f-cbb7b7e18392 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:19.241726 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4bcfe00-8445-4135-af1f-cbb7b7e18392 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:19.247713 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 530/1050] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:43:19 2022] GET /identity/v3/services?type=image => generated 350 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:43:19.253551 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767d6571-1abd-42ee-acf0-49df143366f9 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:19.331990 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:19.373725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767d6571-1abd-42ee-acf0-49df143366f9 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:19.380367 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b5e399-f907-42d5-aacd-522f65858631 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:19.380866 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b5e399-f907-42d5-aacd-522f65858631 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:19.381285 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b5e399-f907-42d5-aacd-522f65858631 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:19.382657 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 531/1051] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:43:19 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 00:43:19.387796 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd0ebc1-108c-4c22-bf2b-fea9b9ae7c5d None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:19.388076 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd0ebc1-108c-4c22-bf2b-fea9b9ae7c5d None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:19.388346 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd0ebc1-108c-4c22-bf2b-fea9b9ae7c5d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:19.407711 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-9fd0ebc1-108c-4c22-bf2b-fea9b9ae7c5d None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:43:19.408892 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-9fd0ebc1-108c-4c22-bf2b-fea9b9ae7c5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:43:19.416243 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e443b5b-3442-4a75-9e10-137d4b174a85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:19.417000 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e443b5b-3442-4a75-9e10-137d4b174a85 None admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:19.417425 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e443b5b-3442-4a75-9e10-137d4b174a85 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:19.417699 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e443b5b-3442-4a75-9e10-137d4b174a85 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:19.418687 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e443b5b-3442-4a75-9e10-137d4b174a85 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:19.420476 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e443b5b-3442-4a75-9e10-137d4b174a85 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:19.440967 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 521/1052] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:43:19 2022] POST /identity/v3/endpoints => generated 327 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:43:19.474895 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:19.507893 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 532/1053] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:43:19 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 25 00:43:19.777668 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e443b5b-3442-4a75-9e10-137d4b174a85 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:19.884542 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e443b5b-3442-4a75-9e10-137d4b174a85 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:19.909125 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:19.922299 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d97b02-faae-44c3-b603-4bf9c998f72d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:19.922941 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d97b02-faae-44c3-b603-4bf9c998f72d None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:19.923216 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d97b02-faae-44c3-b603-4bf9c998f72d None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:19.923484 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d97b02-faae-44c3-b603-4bf9c998f72d None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:19.934446 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44d97b02-faae-44c3-b603-4bf9c998f72d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:43:19.939304 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-44d97b02-faae-44c3-b603-4bf9c998f72d None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 25 00:43:19.939304 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:19.939304 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:43:19.939304 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:43:19.939304 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:43:19.939304 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:43:19.939304 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:43:19.939304 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:43:19.939304 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:43:19.939304 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:43:19.939304 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:43:19.939304 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:43:19.939304 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:43:19.939304 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:43:19.939304 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:43:19.939304 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:43:19.940269 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:43:19.940269 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:43:19.940269 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:43:19.940269 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:43:19.940269 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:43:19.940269 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:43:19.940269 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:43:19.940269 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:43:19.940269 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:43:19.940269 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:43:19.940269 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:43:19.940269 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:43:19.940269 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:43:19.940269 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:43:19.940269 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:43:19.940269 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:43:19.941152 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:43:19.941152 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:43:19.941152 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:43:19.941152 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:43:19.941152 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:43:19.941152 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:43:19.941152 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:43:19.941152 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:43:19.941152 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:43:19.941152 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:43:19.941152 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 25 00:43:19.941152 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:43:19.945343 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 522/1054] 10.222.0.52 () {56 vars in 1120 bytes} [Sun Sep 25 00:43:19 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:43:19.947488 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd0ebc1-108c-4c22-bf2b-fea9b9ae7c5d None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:20.057824 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd0ebc1-108c-4c22-bf2b-fea9b9ae7c5d None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:20.099328 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c175100-11e5-40b4-a084-1a194c517cd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:20.100163 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c175100-11e5-40b4-a084-1a194c517cd3 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:20.100432 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c175100-11e5-40b4-a084-1a194c517cd3 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:20.100696 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c175100-11e5-40b4-a084-1a194c517cd3 None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:20.101418 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c175100-11e5-40b4-a084-1a194c517cd3 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:20.101721 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c175100-11e5-40b4-a084-1a194c517cd3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:20.103113 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c175100-11e5-40b4-a084-1a194c517cd3 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:20.116689 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 533/1055] 10.222.0.52 () {56 vars in 1115 bytes} [Sun Sep 25 00:43:19 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:20.118333 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d97b02-faae-44c3-b603-4bf9c998f72d None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:20.193577 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bac5dc5-65df-4396-8e0f-1d6a813128e1 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:20.193997 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bac5dc5-65df-4396-8e0f-1d6a813128e1 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:20.194143 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bac5dc5-65df-4396-8e0f-1d6a813128e1 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:20.195639 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 534/1056] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:43:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 00:43:20.200619 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7603c8f-9af6-4617-9acf-03ef49e863b4 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:20.200930 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7603c8f-9af6-4617-9acf-03ef49e863b4 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:20.201214 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7603c8f-9af6-4617-9acf-03ef49e863b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:20.220273 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-d7603c8f-9af6-4617-9acf-03ef49e863b4 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:43:20.221503 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-d7603c8f-9af6-4617-9acf-03ef49e863b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:43:20.234265 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d97b02-faae-44c3-b603-4bf9c998f72d None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:20.256541 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:20.272463 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-495a20b6-5675-4f1b-a2e3-eeebaba1f36f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:20.273480 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495a20b6-5675-4f1b-a2e3-eeebaba1f36f None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:20.273961 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495a20b6-5675-4f1b-a2e3-eeebaba1f36f None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:20.274412 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495a20b6-5675-4f1b-a2e3-eeebaba1f36f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:20.286533 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-495a20b6-5675-4f1b-a2e3-eeebaba1f36f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:20.288439 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-495a20b6-5675-4f1b-a2e3-eeebaba1f36f None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:20.295602 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 535/1057] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:43:20 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 25 00:43:20.303883 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-495a20b6-5675-4f1b-a2e3-eeebaba1f36f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:43:20.303883 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:20.303883 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:43:20.303883 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:43:20.303883 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:43:20.303883 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:43:20.303883 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:43:20.303883 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:43:20.303883 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:43:20.303883 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:43:20.303883 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:43:20.303883 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:43:20.303883 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:43:20.303883 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:43:20.303883 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:43:20.303883 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:43:20.304869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:43:20.304869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:43:20.304869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:43:20.304869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:43:20.304869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:43:20.304869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:43:20.304869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:43:20.304869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:43:20.304869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:43:20.304869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:43:20.304869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:43:20.304869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:43:20.304869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:43:20.304869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:43:20.304869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:43:20.304869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:43:20.304869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:43:20.310392 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:43:20.310392 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:43:20.310392 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:43:20.310392 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 523/1058] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:43:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:43:20.311103 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7603c8f-9af6-4617-9acf-03ef49e863b4 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:20.418472 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7603c8f-9af6-4617-9acf-03ef49e863b4 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:20.453882 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-885f56e3-ffba-4743-bc37-b4a09281f100 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:20.454482 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885f56e3-ffba-4743-bc37-b4a09281f100 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:20.454753 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885f56e3-ffba-4743-bc37-b4a09281f100 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:20.455019 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885f56e3-ffba-4743-bc37-b4a09281f100 None admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:20.455657 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-885f56e3-ffba-4743-bc37-b4a09281f100 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:20.455958 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-885f56e3-ffba-4743-bc37-b4a09281f100 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:20.457417 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-885f56e3-ffba-4743-bc37-b4a09281f100 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:20.471316 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 536/1059] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:43:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:20.475579 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-495a20b6-5675-4f1b-a2e3-eeebaba1f36f None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:20.518993 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:20.569865 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-885f56e3-ffba-4743-bc37-b4a09281f100 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:20.588787 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-495a20b6-5675-4f1b-a2e3-eeebaba1f36f None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:20.598852 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:20.627098 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f0f2d0-bd13-4789-8689-bf3fbb2f7dcd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:20.627726 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f0f2d0-bd13-4789-8689-bf3fbb2f7dcd None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:20.628011 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f0f2d0-bd13-4789-8689-bf3fbb2f7dcd None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:20.628279 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f0f2d0-bd13-4789-8689-bf3fbb2f7dcd None admin] PATH_INFO: `/v3/users/nova` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:20.637126 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82f0f2d0-bd13-4789-8689-bf3fbb2f7dcd None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:20.638700 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82f0f2d0-bd13-4789-8689-bf3fbb2f7dcd None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:20.650326 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-82f0f2d0-bd13-4789-8689-bf3fbb2f7dcd None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 25 00:43:20.650326 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:20.650326 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:43:20.650326 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:43:20.650326 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:43:20.650326 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:43:20.650326 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:43:20.650326 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:43:20.650326 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:43:20.650326 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:43:20.650326 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:43:20.650326 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:43:20.650326 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 00:43:20.650326 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 00:43:20.650326 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 00:43:20.650326 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 00:43:20.651370 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:43:20.651370 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:43:20.651370 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 00:43:20.651370 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:43:20.651370 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 00:43:20.651370 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:43:20.651370 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:43:20.651370 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:43:20.651370 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:43:20.651370 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:43:20.651370 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:43:20.651370 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:43:20.651370 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:43:20.651370 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:43:20.651370 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:43:20.651370 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:43:20.651370 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:43:20.651370 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:43:20.652353 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:43:20.652353 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:43:20.652353 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 00:43:20.652353 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 00:43:20.652353 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 00:43:20.652353 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 00:43:20.652353 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 00:43:20.652353 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 00:43:20.652353 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 25 00:43:20.652353 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:43:20.652353 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 524/1060] 10.222.0.52 () {56 vars in 1093 bytes} [Sun Sep 25 00:43:20 2022] GET /identity/v3/users/nova => generated 82 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:43:20.657410 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f0f2d0-bd13-4789-8689-bf3fbb2f7dcd None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:20.685019 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-885f56e3-ffba-4743-bc37-b4a09281f100 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:20.726102 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4351f3a0-2953-41c4-9755-a889ba3ea5a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:20.727088 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4351f3a0-2953-41c4-9755-a889ba3ea5a9 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:20.727619 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4351f3a0-2953-41c4-9755-a889ba3ea5a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:20.728092 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4351f3a0-2953-41c4-9755-a889ba3ea5a9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:20.740413 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4351f3a0-2953-41c4-9755-a889ba3ea5a9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:43:20.745647 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-4351f3a0-2953-41c4-9755-a889ba3ea5a9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:43:20.745647 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:20.745647 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:43:20.745647 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:43:20.745647 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:43:20.745647 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:43:20.745647 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:43:20.745647 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:43:20.745647 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:43:20.745647 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:43:20.745647 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:43:20.745647 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:43:20.745647 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:43:20.745647 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:43:20.745647 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:43:20.745647 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:43:20.746865 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:43:20.746865 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:43:20.746865 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:43:20.746865 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:43:20.746865 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:43:20.746865 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:43:20.746865 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:43:20.746865 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:43:20.746865 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:43:20.746865 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:43:20.746865 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:43:20.746865 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:43:20.746865 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:43:20.746865 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:43:20.746865 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:43:20.746865 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:43:20.747925 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:43:20.747925 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:43:20.747925 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:43:20.747925 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:43:20.747925 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:43:20.747925 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:43:20.747925 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:43:20.747925 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:43:20.747925 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:43:20.747925 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:43:20.747925 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:43:20.747925 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:43:20.749969 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 537/1061] 10.222.0.52 () {56 vars in 1096 bytes} [Sun Sep 25 00:43:20 2022] GET /identity/v3/roles/admin => generated 83 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:43:20.756749 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4351f3a0-2953-41c4-9755-a889ba3ea5a9 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:20.766647 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:20.774331 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f0f2d0-bd13-4789-8689-bf3fbb2f7dcd None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:20.809990 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d33b859-be48-443b-aed3-91f8dde91a2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:20.810603 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d33b859-be48-443b-aed3-91f8dde91a2a None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:20.810880 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d33b859-be48-443b-aed3-91f8dde91a2a None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:20.811149 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d33b859-be48-443b-aed3-91f8dde91a2a None admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:20.811878 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d33b859-be48-443b-aed3-91f8dde91a2a None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:20.812236 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d33b859-be48-443b-aed3-91f8dde91a2a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:20.813718 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d33b859-be48-443b-aed3-91f8dde91a2a None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:20.837682 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 525/1062] 10.222.0.52 () {56 vars in 1133 bytes} [Sun Sep 25 00:43:20 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:43:20.838640 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d33b859-be48-443b-aed3-91f8dde91a2a None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:20.852312 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:20.867538 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4351f3a0-2953-41c4-9755-a889ba3ea5a9 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:20.908061 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec88017c-e664-48d7-a95c-373e13ad3dfb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:20.908732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec88017c-e664-48d7-a95c-373e13ad3dfb None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:20.909003 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec88017c-e664-48d7-a95c-373e13ad3dfb None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:20.909474 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec88017c-e664-48d7-a95c-373e13ad3dfb None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:20.910020 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec88017c-e664-48d7-a95c-373e13ad3dfb None admin] RBAC: Adding query filter params (name=admin) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:20.910317 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec88017c-e664-48d7-a95c-373e13ad3dfb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:20.912549 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec88017c-e664-48d7-a95c-373e13ad3dfb None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:20.928958 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 538/1063] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:43:20 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:43:20.930248 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec88017c-e664-48d7-a95c-373e13ad3dfb None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:20.955012 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d33b859-be48-443b-aed3-91f8dde91a2a None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:20.990468 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a588124d-6d27-4cc5-8357-b7659506eeae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:20.991101 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a588124d-6d27-4cc5-8357-b7659506eeae None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:20.991378 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a588124d-6d27-4cc5-8357-b7659506eeae None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:20.991645 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a588124d-6d27-4cc5-8357-b7659506eeae None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:21.001927 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a588124d-6d27-4cc5-8357-b7659506eeae None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:21.003355 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a588124d-6d27-4cc5-8357-b7659506eeae None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:21.019359 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-a588124d-6d27-4cc5-8357-b7659506eeae None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:43:21.019359 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:21.019359 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:43:21.019359 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:43:21.019359 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:43:21.019359 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:43:21.019359 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:43:21.019359 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:43:21.019359 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:43:21.019359 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:43:21.019359 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:43:21.019359 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:43:21.019359 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:43:21.019359 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:43:21.019359 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:43:21.019359 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:43:21.020457 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:43:21.020457 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:43:21.020457 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:43:21.020457 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:43:21.020457 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:43:21.020457 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:43:21.020457 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:43:21.020457 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:43:21.020457 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:43:21.020457 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:43:21.020457 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:43:21.020457 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:43:21.020457 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:43:21.020457 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:43:21.020457 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:43:21.020457 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:43:21.020457 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:43:21.021445 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:43:21.021445 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:43:21.021445 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:43:21.021445 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 526/1064] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:43:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:43:21.026217 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a588124d-6d27-4cc5-8357-b7659506eeae None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:21.035210 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:21.040551 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec88017c-e664-48d7-a95c-373e13ad3dfb None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:21.077661 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d286a8a1-e5a4-40a5-9625-06317129cc0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:21.078290 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d286a8a1-e5a4-40a5-9625-06317129cc0c None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:21.078576 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d286a8a1-e5a4-40a5-9625-06317129cc0c None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:21.078828 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d286a8a1-e5a4-40a5-9625-06317129cc0c None admin] PATH_INFO: `/v3/users/17e2a2075cb44c5c98ea296a226f103a` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:21.094485 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d286a8a1-e5a4-40a5-9625-06317129cc0c None admin] RBAC: Authorizing `identity:get_user(user_id=17e2a2075cb44c5c98ea296a226f103a)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:21.095834 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d286a8a1-e5a4-40a5-9625-06317129cc0c None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:21.097512 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 539/1065] 10.222.0.52 () {56 vars in 1177 bytes} [Sun Sep 25 00:43:20 2022] GET /identity/v3/users/17e2a2075cb44c5c98ea296a226f103a => generated 285 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:21.103528 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d286a8a1-e5a4-40a5-9625-06317129cc0c None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:21.114955 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:21.135347 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a588124d-6d27-4cc5-8357-b7659506eeae None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:21.171504 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11bd8728-bae6-492b-a706-9b0075add7ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:21.172102 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bd8728-bae6-492b-a706-9b0075add7ac None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:21.172374 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bd8728-bae6-492b-a706-9b0075add7ac None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:21.172633 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bd8728-bae6-492b-a706-9b0075add7ac None admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:21.173294 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11bd8728-bae6-492b-a706-9b0075add7ac None admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:21.173604 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11bd8728-bae6-492b-a706-9b0075add7ac None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:21.174960 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11bd8728-bae6-492b-a706-9b0075add7ac None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:21.188452 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 527/1066] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:43:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:43:21.193894 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11bd8728-bae6-492b-a706-9b0075add7ac None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:21.215569 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d286a8a1-e5a4-40a5-9625-06317129cc0c None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:21.252789 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f60b6cfe-0c98-4d35-8974-5c42e38f2722 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:21.253421 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60b6cfe-0c98-4d35-8974-5c42e38f2722 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:21.253700 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60b6cfe-0c98-4d35-8974-5c42e38f2722 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:21.253979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60b6cfe-0c98-4d35-8974-5c42e38f2722 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:21.261569 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:21.298795 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11bd8728-bae6-492b-a706-9b0075add7ac None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:21.331117 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f60b6cfe-0c98-4d35-8974-5c42e38f2722 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:21.332553 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f60b6cfe-0c98-4d35-8974-5c42e38f2722 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:21.334611 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b77ea44d-688b-40a4-a23a-03ec4b3e413b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:21.335247 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77ea44d-688b-40a4-a23a-03ec4b3e413b None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:21.335527 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77ea44d-688b-40a4-a23a-03ec4b3e413b None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:21.335799 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77ea44d-688b-40a4-a23a-03ec4b3e413b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:21.342757 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b77ea44d-688b-40a4-a23a-03ec4b3e413b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:21.343388 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:21.344600 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b77ea44d-688b-40a4-a23a-03ec4b3e413b None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:21.357796 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-b77ea44d-688b-40a4-a23a-03ec4b3e413b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:43:21.357796 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:21.357796 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:43:21.357796 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:43:21.357796 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:43:21.357796 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:43:21.357796 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:43:21.357796 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:43:21.357796 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:43:21.357796 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:43:21.357796 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:43:21.357796 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:43:21.357796 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 00:43:21.357796 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 00:43:21.357796 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 00:43:21.358736 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 00:43:21.358736 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:43:21.358736 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:43:21.358736 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:43:21.358736 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:43:21.358736 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:43:21.358736 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:43:21.358736 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:43:21.358736 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:43:21.358736 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:43:21.358736 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:43:21.358736 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:43:21.358736 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:43:21.358736 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:43:21.358736 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:43:21.358736 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:43:21.358736 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:43:21.359657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 00:43:21.359657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 00:43:21.359657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 00:43:21.359657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 00:43:21.359657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 00:43:21.359657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 00:43:21.359657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:43:21.359657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:43:21.359657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 528/1067] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:43:21 2022] GET /identity/v3/projects/service => generated 88 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:43:21.366288 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b77ea44d-688b-40a4-a23a-03ec4b3e413b None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:21.446923 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 540/1068] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:43:21 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 344 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 25 00:43:21.452470 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f60b6cfe-0c98-4d35-8974-5c42e38f2722 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:21.474046 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b77ea44d-688b-40a4-a23a-03ec4b3e413b None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:21.505713 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:21.516995 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a84bf67-87e0-4303-abc0-24abb0e9372b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:21.517655 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a84bf67-87e0-4303-abc0-24abb0e9372b None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:21.517934 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a84bf67-87e0-4303-abc0-24abb0e9372b None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:21.518203 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a84bf67-87e0-4303-abc0-24abb0e9372b None admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:21.518894 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a84bf67-87e0-4303-abc0-24abb0e9372b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:21.519196 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a84bf67-87e0-4303-abc0-24abb0e9372b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:21.520863 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a84bf67-87e0-4303-abc0-24abb0e9372b None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:21.533481 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 529/1069] 10.222.0.52 () {56 vars in 1148 bytes} [Sun Sep 25 00:43:21 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:21.540354 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a84bf67-87e0-4303-abc0-24abb0e9372b None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:21.580017 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f60b6cfe-0c98-4d35-8974-5c42e38f2722 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:21.591090 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:21.625004 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1375c44d-e7d3-4a3a-a917-09fadc05e3fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:21.625576 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1375c44d-e7d3-4a3a-a917-09fadc05e3fc None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:21.625863 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1375c44d-e7d3-4a3a-a917-09fadc05e3fc None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:21.626114 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1375c44d-e7d3-4a3a-a917-09fadc05e3fc None admin] PATH_INFO: `/v3/projects/36a6c7219a784408a5462f30fa36d521` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:21.627708 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1375c44d-e7d3-4a3a-a917-09fadc05e3fc None admin] RBAC: Authorizing `identity:get_project(project_id=36a6c7219a784408a5462f30fa36d521)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:21.630374 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1375c44d-e7d3-4a3a-a917-09fadc05e3fc None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:21.632841 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 541/1070] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:43:21 2022] GET /identity/v3/projects/36a6c7219a784408a5462f30fa36d521 => generated 311 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:43:21.638615 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1375c44d-e7d3-4a3a-a917-09fadc05e3fc None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:21.655317 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a84bf67-87e0-4303-abc0-24abb0e9372b None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:21.700300 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b467aea0-228b-492f-880e-2b51fd7d2f45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:21.701188 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b467aea0-228b-492f-880e-2b51fd7d2f45 None admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:21.701530 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b467aea0-228b-492f-880e-2b51fd7d2f45 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:21.701806 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b467aea0-228b-492f-880e-2b51fd7d2f45 None admin] PATH_INFO: `/v3/projects/9db0fd07fc494203b2ada13f3cd046d3/users/f142988a4dd14a5f96eacea45c046f85/roles/5d162233989d4e439c441608704dc9b0` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:21.704442 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b467aea0-228b-492f-880e-2b51fd7d2f45 None admin] RBAC: Authorizing `identity:create_grant(project_id=9db0fd07fc494203b2ada13f3cd046d3, user_id=f142988a4dd14a5f96eacea45c046f85, role_id=5d162233989d4e439c441608704dc9b0)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:21.706099 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b467aea0-228b-492f-880e-2b51fd7d2f45 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:21.721599 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 530/1071] 10.222.0.52 () {58 vars in 1439 bytes} [Sun Sep 25 00:43:21 2022] PUT /identity/v3/projects/9db0fd07fc494203b2ada13f3cd046d3/users/f142988a4dd14a5f96eacea45c046f85/roles/5d162233989d4e439c441608704dc9b0 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 00:43:21.727523 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fdff684-2aca-4108-a3b8-003d83549f45 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:21.727861 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fdff684-2aca-4108-a3b8-003d83549f45 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:21.728340 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fdff684-2aca-4108-a3b8-003d83549f45 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:21.730243 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 531/1072] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:43:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 00:43:21.738196 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48fca773-6e5b-4e51-8ff2-69a8b95c2e1d None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:21.738430 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48fca773-6e5b-4e51-8ff2-69a8b95c2e1d None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:21.738794 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48fca773-6e5b-4e51-8ff2-69a8b95c2e1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:21.760960 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-48fca773-6e5b-4e51-8ff2-69a8b95c2e1d None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:43:21.762281 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-48fca773-6e5b-4e51-8ff2-69a8b95c2e1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:43:21.768300 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1375c44d-e7d3-4a3a-a917-09fadc05e3fc None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:21.812654 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641996a4-b8db-441e-8528-d85a7ff25562 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:21.813280 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641996a4-b8db-441e-8528-d85a7ff25562 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:21.813576 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641996a4-b8db-441e-8528-d85a7ff25562 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:21.813847 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641996a4-b8db-441e-8528-d85a7ff25562 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:21.814579 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-641996a4-b8db-441e-8528-d85a7ff25562 None admin] RBAC: Adding query filter params (role.id=2c242abf6ed8410ab0a4a354ca1794fb, scope.project.id=36a6c7219a784408a5462f30fa36d521, user.id=17e2a2075cb44c5c98ea296a226f103a) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:21.814882 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-641996a4-b8db-441e-8528-d85a7ff25562 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:21.816345 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-641996a4-b8db-441e-8528-d85a7ff25562 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:21.818830 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:21.824409 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 542/1073] 10.222.0.52 () {56 vars in 1374 bytes} [Sun Sep 25 00:43:21 2022] GET /identity/v3/role_assignments?user.id=17e2a2075cb44c5c98ea296a226f103a&scope.project.id=36a6c7219a784408a5462f30fa36d521&role.id=2c242abf6ed8410ab0a4a354ca1794fb => generated 259 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:43:21.852952 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 532/1074] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:43:21 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 25 00:43:22.124373 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641996a4-b8db-441e-8528-d85a7ff25562 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:22.169648 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:22.232632 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641996a4-b8db-441e-8528-d85a7ff25562 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:22.268841 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e081662-bdca-4465-84f4-e07bae62bc45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:22.269489 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e081662-bdca-4465-84f4-e07bae62bc45 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:22.269761 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e081662-bdca-4465-84f4-e07bae62bc45 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:22.270020 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e081662-bdca-4465-84f4-e07bae62bc45 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:22.341468 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e081662-bdca-4465-84f4-e07bae62bc45 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:22.342907 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e081662-bdca-4465-84f4-e07bae62bc45 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:22.385921 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:22.447329 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 543/1075] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:43:22 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 25 00:43:22.454955 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48fca773-6e5b-4e51-8ff2-69a8b95c2e1d None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:22.568544 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48fca773-6e5b-4e51-8ff2-69a8b95c2e1d None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:22.603244 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2986d663-1180-4445-9c1f-964e3a04c075 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:22.603873 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2986d663-1180-4445-9c1f-964e3a04c075 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:22.604148 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2986d663-1180-4445-9c1f-964e3a04c075 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:22.604415 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2986d663-1180-4445-9c1f-964e3a04c075 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:22.616049 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2986d663-1180-4445-9c1f-964e3a04c075 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:22.617536 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2986d663-1180-4445-9c1f-964e3a04c075 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:22.633596 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-2986d663-1180-4445-9c1f-964e3a04c075 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:43:22.633596 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:22.633596 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:43:22.633596 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:43:22.633596 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:43:22.633596 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:43:22.633596 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:43:22.633596 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:43:22.633596 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:43:22.633596 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:43:22.633596 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:43:22.633596 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:43:22.633596 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:43:22.633596 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:43:22.633596 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:43:22.633596 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:43:22.634674 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:43:22.634674 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:43:22.634674 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:43:22.634674 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:43:22.634674 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:43:22.634674 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:43:22.634674 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:43:22.634674 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:43:22.634674 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:43:22.634674 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:43:22.634674 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:43:22.634674 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:43:22.634674 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:43:22.634674 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:43:22.634674 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:43:22.634674 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:43:22.634674 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:43:22.635583 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:43:22.635583 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:43:22.635583 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:43:22.636611 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 533/1076] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:43:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:43:22.640964 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e081662-bdca-4465-84f4-e07bae62bc45 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:22.762290 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e081662-bdca-4465-84f4-e07bae62bc45 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:22.801354 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f55cdd0-26df-446b-88ed-e8cd816b28b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:22.802036 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f55cdd0-26df-446b-88ed-e8cd816b28b5 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:22.802310 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f55cdd0-26df-446b-88ed-e8cd816b28b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:22.802609 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f55cdd0-26df-446b-88ed-e8cd816b28b5 None admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:22.803205 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f55cdd0-26df-446b-88ed-e8cd816b28b5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:22.803502 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f55cdd0-26df-446b-88ed-e8cd816b28b5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:22.804990 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f55cdd0-26df-446b-88ed-e8cd816b28b5 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:22.813717 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:22.821468 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 544/1077] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:43:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:23.968672 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7be891-3faa-41e8-acd9-a73ee9cd9442 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:23.969179 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7be891-3faa-41e8-acd9-a73ee9cd9442 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:23.969179 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7be891-3faa-41e8-acd9-a73ee9cd9442 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:23.970493 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 534/1078] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:43:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:43:23.976279 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4afeb17-140a-4a94-b51f-1c335e682c3f None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:23.976686 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4afeb17-140a-4a94-b51f-1c335e682c3f None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:23.976808 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4afeb17-140a-4a94-b51f-1c335e682c3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:23.996339 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-e4afeb17-140a-4a94-b51f-1c335e682c3f None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:43:23.997527 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-e4afeb17-140a-4a94-b51f-1c335e682c3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:43:24.022578 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6ef90c-8638-4e08-bb63-910291720c5f None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:24.022863 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6ef90c-8638-4e08-bb63-910291720c5f None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:24.023135 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6ef90c-8638-4e08-bb63-910291720c5f None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:24.024449 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 535/1079] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:43:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:43:24.029428 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a405812f-b38d-4706-803c-76d113ec415e None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:24.029778 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a405812f-b38d-4706-803c-76d113ec415e None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:24.030117 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a405812f-b38d-4706-803c-76d113ec415e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:24.048680 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-a405812f-b38d-4706-803c-76d113ec415e None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:43:24.049992 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-a405812f-b38d-4706-803c-76d113ec415e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:43:24.069181 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 545/1080] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:43:23 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:43:24.073117 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:24.117750 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 536/1081] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:43:24 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 25 00:43:24.356974 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4afeb17-140a-4a94-b51f-1c335e682c3f None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:24.388916 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a405812f-b38d-4706-803c-76d113ec415e None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:24.459705 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4afeb17-140a-4a94-b51f-1c335e682c3f None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:24.489749 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:24.494288 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a405812f-b38d-4706-803c-76d113ec415e None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:24.498352 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-624fc146-99d6-4651-ba9f-55236841080d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:24.498352 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-624fc146-99d6-4651-ba9f-55236841080d None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:24.498352 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-624fc146-99d6-4651-ba9f-55236841080d None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:24.498352 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-624fc146-99d6-4651-ba9f-55236841080d None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:24.509552 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-624fc146-99d6-4651-ba9f-55236841080d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:43:24.514438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-624fc146-99d6-4651-ba9f-55236841080d None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 25 00:43:24.514438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:24.514438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:43:24.514438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:43:24.514438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:43:24.514438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:43:24.514438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:43:24.514438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:43:24.514438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:43:24.514438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:43:24.514438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:43:24.514438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:43:24.514438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:43:24.514438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:43:24.514438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:43:24.514438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:43:24.515614 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:43:24.515614 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:43:24.515614 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:43:24.515614 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:43:24.515614 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:43:24.515614 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:43:24.515614 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:43:24.515614 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:43:24.515614 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:43:24.515614 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:43:24.515614 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:43:24.515614 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:43:24.515614 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:43:24.515614 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:43:24.515614 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:43:24.515614 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:43:24.516600 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:43:24.516600 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:43:24.516600 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:43:24.516600 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:43:24.516600 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:43:24.516600 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:43:24.516600 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:43:24.516600 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:43:24.516600 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:43:24.516600 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:43:24.516600 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 25 00:43:24.516600 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:43:24.516600 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 546/1082] 10.222.0.52 () {56 vars in 1120 bytes} [Sun Sep 25 00:43:24 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:43:24.521800 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-624fc146-99d6-4651-ba9f-55236841080d None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:24.532013 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b017510-12f5-434b-8d3a-13892ef4fc63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:24.532642 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b017510-12f5-434b-8d3a-13892ef4fc63 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:24.532910 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b017510-12f5-434b-8d3a-13892ef4fc63 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:24.533173 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b017510-12f5-434b-8d3a-13892ef4fc63 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:24.543631 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b017510-12f5-434b-8d3a-13892ef4fc63 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:43:24.548391 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-9b017510-12f5-434b-8d3a-13892ef4fc63 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:43:24.548391 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:24.548391 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:43:24.548391 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:43:24.548391 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:43:24.548391 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:43:24.548391 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:43:24.548391 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:43:24.548391 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:43:24.548391 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:43:24.548391 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:43:24.548391 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:43:24.548391 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:43:24.548391 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:43:24.548391 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:43:24.548391 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:43:24.549555 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:43:24.549555 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:43:24.549555 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:43:24.549555 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:43:24.549555 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:43:24.549555 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:43:24.549555 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:43:24.549555 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:43:24.549555 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:43:24.549555 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:43:24.549555 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:43:24.549555 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:43:24.549555 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:43:24.549555 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:43:24.549555 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:43:24.549555 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:43:24.550538 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:43:24.550538 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:43:24.550538 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:43:24.550538 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:43:24.550538 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:43:24.550538 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:43:24.550538 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:43:24.550538 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:43:24.550538 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:43:24.550538 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:43:24.550538 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:43:24.550538 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:43:24.550538 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 537/1083] 10.222.0.52 () {56 vars in 1096 bytes} [Sun Sep 25 00:43:24 2022] GET /identity/v3/roles/admin => generated 83 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:43:24.556092 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b017510-12f5-434b-8d3a-13892ef4fc63 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:24.575360 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:24.629155 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-624fc146-99d6-4651-ba9f-55236841080d None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:24.664261 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b017510-12f5-434b-8d3a-13892ef4fc63 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:24.665373 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afa80f16-36c8-4977-8a3f-d97bbf5acd4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:24.665984 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa80f16-36c8-4977-8a3f-d97bbf5acd4f None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:24.666255 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa80f16-36c8-4977-8a3f-d97bbf5acd4f None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:24.666535 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa80f16-36c8-4977-8a3f-d97bbf5acd4f None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:24.667199 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afa80f16-36c8-4977-8a3f-d97bbf5acd4f None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:24.667497 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afa80f16-36c8-4977-8a3f-d97bbf5acd4f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:24.668859 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afa80f16-36c8-4977-8a3f-d97bbf5acd4f None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:24.677804 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 547/1084] 10.222.0.52 () {56 vars in 1115 bytes} [Sun Sep 25 00:43:24 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:43:24.684619 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afa80f16-36c8-4977-8a3f-d97bbf5acd4f None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:24.708096 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fba1802-9f70-410f-8055-8b02050f9ba8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:24.709524 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fba1802-9f70-410f-8055-8b02050f9ba8 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:24.709976 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fba1802-9f70-410f-8055-8b02050f9ba8 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:24.710345 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fba1802-9f70-410f-8055-8b02050f9ba8 None admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:24.711374 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fba1802-9f70-410f-8055-8b02050f9ba8 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:24.711715 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fba1802-9f70-410f-8055-8b02050f9ba8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:24.713223 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fba1802-9f70-410f-8055-8b02050f9ba8 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:24.723367 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 538/1085] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:43:24 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:43:24.732261 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fba1802-9f70-410f-8055-8b02050f9ba8 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:24.752076 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:24.805246 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afa80f16-36c8-4977-8a3f-d97bbf5acd4f None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:24.830025 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:24.839888 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26a34a4-d83a-41f1-92ec-5edbb1d3a288 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:24.840514 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26a34a4-d83a-41f1-92ec-5edbb1d3a288 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:24.840789 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26a34a4-d83a-41f1-92ec-5edbb1d3a288 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:24.841052 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26a34a4-d83a-41f1-92ec-5edbb1d3a288 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:24.850074 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fba1802-9f70-410f-8055-8b02050f9ba8 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:24.852083 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e26a34a4-d83a-41f1-92ec-5edbb1d3a288 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:24.853550 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e26a34a4-d83a-41f1-92ec-5edbb1d3a288 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:24.868546 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-e26a34a4-d83a-41f1-92ec-5edbb1d3a288 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:43:24.868546 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:24.868546 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:43:24.868546 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:43:24.868546 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:43:24.868546 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:43:24.868546 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:43:24.868546 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:43:24.868546 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:43:24.868546 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:43:24.868546 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:43:24.868546 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:43:24.868546 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:43:24.868546 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:43:24.868546 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:43:24.868546 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:43:24.869736 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:43:24.869736 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:43:24.869736 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:43:24.869736 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:43:24.869736 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:43:24.869736 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:43:24.869736 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:43:24.869736 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:43:24.869736 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:43:24.869736 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:43:24.869736 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:43:24.869736 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:43:24.869736 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:43:24.869736 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:43:24.869736 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:43:24.869736 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:43:24.869736 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:43:24.871253 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:43:24.871253 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:43:24.871253 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:43:24.871253 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 548/1086] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:43:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:43:24.875335 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26a34a4-d83a-41f1-92ec-5edbb1d3a288 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:24.886466 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4f58f9-7787-44bc-abbb-38ed02628900 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:24.887024 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4f58f9-7787-44bc-abbb-38ed02628900 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:24.887302 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4f58f9-7787-44bc-abbb-38ed02628900 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:24.887576 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4f58f9-7787-44bc-abbb-38ed02628900 None admin] PATH_INFO: `/v3/users/17e2a2075cb44c5c98ea296a226f103a` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:24.888715 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b4f58f9-7787-44bc-abbb-38ed02628900 None admin] RBAC: Authorizing `identity:get_user(user_id=17e2a2075cb44c5c98ea296a226f103a)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:24.890168 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b4f58f9-7787-44bc-abbb-38ed02628900 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:24.891756 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 539/1087] 10.222.0.52 () {56 vars in 1177 bytes} [Sun Sep 25 00:43:24 2022] GET /identity/v3/users/17e2a2075cb44c5c98ea296a226f103a => generated 285 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:24.897764 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4f58f9-7787-44bc-abbb-38ed02628900 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:24.978184 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26a34a4-d83a-41f1-92ec-5edbb1d3a288 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:24.980446 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:24.998522 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4f58f9-7787-44bc-abbb-38ed02628900 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:25.013773 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b9d2b6c-fbaa-468a-aa50-02b0697ad995 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:25.014452 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9d2b6c-fbaa-468a-aa50-02b0697ad995 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:25.014801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9d2b6c-fbaa-468a-aa50-02b0697ad995 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:25.015108 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9d2b6c-fbaa-468a-aa50-02b0697ad995 None admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:25.015745 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b9d2b6c-fbaa-468a-aa50-02b0697ad995 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:25.016042 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b9d2b6c-fbaa-468a-aa50-02b0697ad995 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:25.017419 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b9d2b6c-fbaa-468a-aa50-02b0697ad995 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:25.029651 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 549/1088] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:43:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:43:25.035002 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63eeee9e-0e3e-44d7-a0ed-ba6c3bea8240 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:25.035176 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b9d2b6c-fbaa-468a-aa50-02b0697ad995 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:25.036107 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63eeee9e-0e3e-44d7-a0ed-ba6c3bea8240 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:25.036384 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63eeee9e-0e3e-44d7-a0ed-ba6c3bea8240 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:25.036653 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63eeee9e-0e3e-44d7-a0ed-ba6c3bea8240 None admin] PATH_INFO: `/v3/projects/36a6c7219a784408a5462f30fa36d521` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:25.037793 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63eeee9e-0e3e-44d7-a0ed-ba6c3bea8240 None admin] RBAC: Authorizing `identity:get_project(project_id=36a6c7219a784408a5462f30fa36d521)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:25.039222 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63eeee9e-0e3e-44d7-a0ed-ba6c3bea8240 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:25.040911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 540/1089] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:43:24 2022] GET /identity/v3/projects/36a6c7219a784408a5462f30fa36d521 => generated 311 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:43:25.046867 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63eeee9e-0e3e-44d7-a0ed-ba6c3bea8240 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:25.066244 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:25.139832 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b9d2b6c-fbaa-468a-aa50-02b0697ad995 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:25.157980 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63eeee9e-0e3e-44d7-a0ed-ba6c3bea8240 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:25.174134 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7960ab92-3469-4b41-b5e9-e0b07440409f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:25.174748 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7960ab92-3469-4b41-b5e9-e0b07440409f None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:25.175020 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7960ab92-3469-4b41-b5e9-e0b07440409f None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:25.175281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7960ab92-3469-4b41-b5e9-e0b07440409f None admin] PATH_INFO: `/v3/users/nova` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:25.183094 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7960ab92-3469-4b41-b5e9-e0b07440409f None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:25.184501 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7960ab92-3469-4b41-b5e9-e0b07440409f None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:25.192828 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee81b365-c522-41c8-a3d5-2b30f18d253b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:25.193517 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee81b365-c522-41c8-a3d5-2b30f18d253b None admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:25.193797 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee81b365-c522-41c8-a3d5-2b30f18d253b None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:25.194069 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee81b365-c522-41c8-a3d5-2b30f18d253b None admin] PATH_INFO: `/v3/projects/36a6c7219a784408a5462f30fa36d521/users/17e2a2075cb44c5c98ea296a226f103a/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:25.195780 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-7960ab92-3469-4b41-b5e9-e0b07440409f None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 25 00:43:25.195780 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:25.195780 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:43:25.195780 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:43:25.195780 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:43:25.195780 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:43:25.195780 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:43:25.195780 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:43:25.195780 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:43:25.195780 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:43:25.195780 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:43:25.195780 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:43:25.195780 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 00:43:25.195780 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 00:43:25.195780 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 00:43:25.195780 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 00:43:25.196861 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:43:25.196861 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:43:25.196861 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 00:43:25.196861 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:43:25.196861 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 00:43:25.196861 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:43:25.196861 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:43:25.196861 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:43:25.196861 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:43:25.196861 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:43:25.196861 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:43:25.196861 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:43:25.196861 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:43:25.196861 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:43:25.196861 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:43:25.196861 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:43:25.196861 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:43:25.196861 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:43:25.197870 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:43:25.197870 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:43:25.197870 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 00:43:25.197870 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 00:43:25.197870 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 00:43:25.197870 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 00:43:25.197870 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 00:43:25.197870 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 00:43:25.197870 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 25 00:43:25.197870 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:43:25.198408 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee81b365-c522-41c8-a3d5-2b30f18d253b None admin] RBAC: Authorizing `identity:create_grant(project_id=36a6c7219a784408a5462f30fa36d521, user_id=17e2a2075cb44c5c98ea296a226f103a, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:25.198488 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 550/1090] 10.222.0.52 () {56 vars in 1093 bytes} [Sun Sep 25 00:43:25 2022] GET /identity/v3/users/nova => generated 82 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:43:25.198558 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee81b365-c522-41c8-a3d5-2b30f18d253b None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:25.202668 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7960ab92-3469-4b41-b5e9-e0b07440409f None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:25.211173 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 541/1091] 10.222.0.52 () {58 vars in 1439 bytes} [Sun Sep 25 00:43:25 2022] PUT /identity/v3/projects/36a6c7219a784408a5462f30fa36d521/users/17e2a2075cb44c5c98ea296a226f103a/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 00:43:25.230209 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:25.314063 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7960ab92-3469-4b41-b5e9-e0b07440409f None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:25.350013 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92364b8f-ce9f-4f49-8d28-801ae102c54e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:25.350621 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92364b8f-ce9f-4f49-8d28-801ae102c54e None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:25.350897 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92364b8f-ce9f-4f49-8d28-801ae102c54e None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:25.351161 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92364b8f-ce9f-4f49-8d28-801ae102c54e None admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:25.351887 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92364b8f-ce9f-4f49-8d28-801ae102c54e None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:25.352191 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92364b8f-ce9f-4f49-8d28-801ae102c54e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:25.353671 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92364b8f-ce9f-4f49-8d28-801ae102c54e None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:25.373571 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 551/1092] 10.222.0.52 () {56 vars in 1133 bytes} [Sun Sep 25 00:43:25 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:43:25.379989 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee81b365-c522-41c8-a3d5-2b30f18d253b None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:25.462923 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:25.486618 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee81b365-c522-41c8-a3d5-2b30f18d253b None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:25.524781 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce2f9c73-d4ec-4f94-8ff0-034f8a37ae2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:25.525495 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2f9c73-d4ec-4f94-8ff0-034f8a37ae2a None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:25.525770 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2f9c73-d4ec-4f94-8ff0-034f8a37ae2a None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:25.526035 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2f9c73-d4ec-4f94-8ff0-034f8a37ae2a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:25.595609 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce2f9c73-d4ec-4f94-8ff0-034f8a37ae2a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:25.597076 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce2f9c73-d4ec-4f94-8ff0-034f8a37ae2a None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:25.679007 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:25.702543 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 542/1093] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:43:25 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 25 00:43:25.709071 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92364b8f-ce9f-4f49-8d28-801ae102c54e None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:25.789553 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:25.807573 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92364b8f-ce9f-4f49-8d28-801ae102c54e None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:25.841316 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc60fb02-edee-4637-8ea4-2bb513f2d7fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:25.841940 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc60fb02-edee-4637-8ea4-2bb513f2d7fd None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:25.842212 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc60fb02-edee-4637-8ea4-2bb513f2d7fd None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:25.842483 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc60fb02-edee-4637-8ea4-2bb513f2d7fd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:25.852552 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc60fb02-edee-4637-8ea4-2bb513f2d7fd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:25.854011 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc60fb02-edee-4637-8ea4-2bb513f2d7fd None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:25.869136 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-bc60fb02-edee-4637-8ea4-2bb513f2d7fd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:43:25.869136 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:25.869136 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:43:25.869136 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:43:25.869136 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:43:25.869136 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:43:25.869136 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:43:25.869136 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:43:25.869136 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:43:25.869136 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:43:25.869136 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:43:25.869136 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:43:25.869136 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:43:25.869136 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:43:25.869136 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:43:25.869136 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:43:25.870117 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:43:25.870117 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:43:25.870117 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:43:25.870117 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:43:25.870117 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:43:25.870117 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:43:25.870117 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:43:25.870117 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:43:25.870117 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:43:25.870117 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:43:25.870117 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:43:25.870117 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:43:25.870117 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:43:25.870117 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:43:25.870117 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:43:25.870117 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:43:25.870117 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:43:25.871009 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:43:25.871009 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:43:25.871009 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:43:25.871009 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 552/1094] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:43:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:43:25.875978 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce2f9c73-d4ec-4f94-8ff0-034f8a37ae2a None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:25.976083 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce2f9c73-d4ec-4f94-8ff0-034f8a37ae2a None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:26.009786 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-084883e8-245f-4b3a-be65-f0aaa7d7576e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:26.010436 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084883e8-245f-4b3a-be65-f0aaa7d7576e None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:26.010645 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084883e8-245f-4b3a-be65-f0aaa7d7576e None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:26.010906 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084883e8-245f-4b3a-be65-f0aaa7d7576e None admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:26.011531 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-084883e8-245f-4b3a-be65-f0aaa7d7576e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:26.011827 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-084883e8-245f-4b3a-be65-f0aaa7d7576e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:26.013480 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-084883e8-245f-4b3a-be65-f0aaa7d7576e None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:26.026152 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 543/1095] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:43:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:26.031733 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc60fb02-edee-4637-8ea4-2bb513f2d7fd None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:26.135015 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc60fb02-edee-4637-8ea4-2bb513f2d7fd None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:26.167617 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1437a98b-7046-433a-ae33-e88a27706375 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:26.168222 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1437a98b-7046-433a-ae33-e88a27706375 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:26.168494 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1437a98b-7046-433a-ae33-e88a27706375 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:26.168758 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1437a98b-7046-433a-ae33-e88a27706375 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:26.170795 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:26.176991 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1437a98b-7046-433a-ae33-e88a27706375 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:26.178797 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1437a98b-7046-433a-ae33-e88a27706375 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:26.189769 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-1437a98b-7046-433a-ae33-e88a27706375 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:43:26.189769 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:26.189769 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:43:26.189769 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:43:26.189769 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:43:26.189769 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:43:26.189769 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:43:26.189769 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:43:26.189769 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:43:26.189769 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:43:26.189769 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:43:26.189769 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:43:26.189769 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 00:43:26.189769 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 00:43:26.189769 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 00:43:26.190802 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 00:43:26.190802 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:43:26.190802 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:43:26.190802 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:43:26.190802 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:43:26.190802 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:43:26.190802 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:43:26.190802 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:43:26.190802 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:43:26.190802 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:43:26.190802 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:43:26.190802 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:43:26.190802 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:43:26.190802 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:43:26.190802 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:43:26.190802 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:43:26.190802 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:43:26.191737 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 00:43:26.191737 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 00:43:26.191737 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 00:43:26.191737 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 00:43:26.191737 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 00:43:26.191737 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 00:43:26.191737 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:43:26.191737 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:43:26.191737 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 553/1096] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:43:26 2022] GET /identity/v3/projects/service => generated 88 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:43:26.196653 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-084883e8-245f-4b3a-be65-f0aaa7d7576e None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:26.231753 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:26.296333 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-084883e8-245f-4b3a-be65-f0aaa7d7576e None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:26.336784 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144b7a06-f58d-4600-9db0-71c2c9701f8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:26.337830 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144b7a06-f58d-4600-9db0-71c2c9701f8f None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:26.338142 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144b7a06-f58d-4600-9db0-71c2c9701f8f None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:26.338415 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144b7a06-f58d-4600-9db0-71c2c9701f8f None admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:26.339646 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-144b7a06-f58d-4600-9db0-71c2c9701f8f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:26.340202 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-144b7a06-f58d-4600-9db0-71c2c9701f8f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:26.342895 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-144b7a06-f58d-4600-9db0-71c2c9701f8f None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:26.356141 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 544/1097] 10.222.0.52 () {56 vars in 1148 bytes} [Sun Sep 25 00:43:26 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:43:26.361961 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1437a98b-7046-433a-ae33-e88a27706375 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:26.461903 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1437a98b-7046-433a-ae33-e88a27706375 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:26.500356 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de3edbbb-4177-4ee6-ac0b-9cd07cad2264 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:26.500356 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de3edbbb-4177-4ee6-ac0b-9cd07cad2264 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:26.500356 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de3edbbb-4177-4ee6-ac0b-9cd07cad2264 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:26.500356 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de3edbbb-4177-4ee6-ac0b-9cd07cad2264 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:26.500769 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de3edbbb-4177-4ee6-ac0b-9cd07cad2264 None admin] RBAC: Adding query filter params (role.id=5d162233989d4e439c441608704dc9b0, scope.project.id=9db0fd07fc494203b2ada13f3cd046d3, user.id=f142988a4dd14a5f96eacea45c046f85) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:26.501025 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de3edbbb-4177-4ee6-ac0b-9cd07cad2264 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:26.502467 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de3edbbb-4177-4ee6-ac0b-9cd07cad2264 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:26.508872 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 554/1098] 10.222.0.52 () {56 vars in 1374 bytes} [Sun Sep 25 00:43:26 2022] GET /identity/v3/role_assignments?user.id=f142988a4dd14a5f96eacea45c046f85&scope.project.id=9db0fd07fc494203b2ada13f3cd046d3&role.id=5d162233989d4e439c441608704dc9b0 => generated 608 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:27.419969 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd969aa8-6b3a-41f1-8d0e-8c2052ab4691 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:27.420501 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd969aa8-6b3a-41f1-8d0e-8c2052ab4691 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:27.420501 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd969aa8-6b3a-41f1-8d0e-8c2052ab4691 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:27.421869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 545/1099] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:43:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:43:27.427894 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a509fa0-80ac-4b3a-ac72-81d51bb8039a None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:27.428381 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a509fa0-80ac-4b3a-ac72-81d51bb8039a None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:27.428503 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a509fa0-80ac-4b3a-ac72-81d51bb8039a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:27.448427 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-2a509fa0-80ac-4b3a-ac72-81d51bb8039a None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:43:27.449650 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-2a509fa0-80ac-4b3a-ac72-81d51bb8039a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:43:27.519143 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:27.533972 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 555/1100] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:43:27 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:43:27.804217 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd969aa8-6b3a-41f1-8d0e-8c2052ab4691 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:27.902018 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:27.909555 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd969aa8-6b3a-41f1-8d0e-8c2052ab4691 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:27.943578 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79bf083c-a955-47d5-8dd5-454c0a781af5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:27.944186 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79bf083c-a955-47d5-8dd5-454c0a781af5 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:27.944457 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79bf083c-a955-47d5-8dd5-454c0a781af5 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:27.944720 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79bf083c-a955-47d5-8dd5-454c0a781af5 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:27.954438 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79bf083c-a955-47d5-8dd5-454c0a781af5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:43:27.959282 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-79bf083c-a955-47d5-8dd5-454c0a781af5 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:43:27.959282 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:27.959282 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:43:27.959282 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:43:27.959282 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:43:27.959282 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:43:27.959282 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:43:27.959282 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:43:27.959282 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:43:27.959282 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:43:27.959282 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:43:27.959282 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:43:27.959282 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:43:27.959282 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:43:27.959282 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:43:27.959282 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:43:27.960416 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:43:27.960416 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:43:27.960416 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:43:27.960416 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:43:27.960416 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:43:27.960416 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:43:27.960416 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:43:27.960416 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:43:27.960416 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:43:27.960416 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:43:27.960416 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:43:27.960416 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:43:27.960416 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:43:27.960416 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:43:27.960416 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:43:27.960416 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:43:27.961799 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:43:27.961799 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:43:27.961799 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:43:27.961799 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:43:27.961799 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:43:27.961799 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:43:27.961799 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:43:27.961799 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:43:27.961799 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:43:27.961799 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:43:27.961799 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:43:27.961799 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:43:27.961799 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 546/1101] 10.222.0.52 () {56 vars in 1096 bytes} [Sun Sep 25 00:43:27 2022] GET /identity/v3/roles/admin => generated 83 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:43:27.966485 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a509fa0-80ac-4b3a-ac72-81d51bb8039a None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:28.062548 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a509fa0-80ac-4b3a-ac72-81d51bb8039a None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:28.100268 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f05d87be-f82f-42e2-b054-a27632ca6d98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:28.100867 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05d87be-f82f-42e2-b054-a27632ca6d98 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:28.101238 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05d87be-f82f-42e2-b054-a27632ca6d98 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:28.101764 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05d87be-f82f-42e2-b054-a27632ca6d98 None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:28.102086 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f05d87be-f82f-42e2-b054-a27632ca6d98 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:28.102388 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f05d87be-f82f-42e2-b054-a27632ca6d98 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:28.103740 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f05d87be-f82f-42e2-b054-a27632ca6d98 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:28.111673 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 556/1102] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:43:27 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:28.118451 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79bf083c-a955-47d5-8dd5-454c0a781af5 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:28.221111 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79bf083c-a955-47d5-8dd5-454c0a781af5 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:28.262180 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e130639-cd62-460e-a22b-4bb0feee374c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:28.262806 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e130639-cd62-460e-a22b-4bb0feee374c None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:28.263079 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e130639-cd62-460e-a22b-4bb0feee374c None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:28.263345 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e130639-cd62-460e-a22b-4bb0feee374c None admin] PATH_INFO: `/v3/users/17e2a2075cb44c5c98ea296a226f103a` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:28.264461 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e130639-cd62-460e-a22b-4bb0feee374c None admin] RBAC: Authorizing `identity:get_user(user_id=17e2a2075cb44c5c98ea296a226f103a)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:28.265923 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e130639-cd62-460e-a22b-4bb0feee374c None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:28.267518 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 547/1103] 10.222.0.52 () {56 vars in 1177 bytes} [Sun Sep 25 00:43:28 2022] GET /identity/v3/users/17e2a2075cb44c5c98ea296a226f103a => generated 285 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:28.273646 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f05d87be-f82f-42e2-b054-a27632ca6d98 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:28.322871 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:28.375456 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f05d87be-f82f-42e2-b054-a27632ca6d98 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:28.408105 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7349c8f9-ad1c-4422-8003-73e3419ad9c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:28.408690 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7349c8f9-ad1c-4422-8003-73e3419ad9c0 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:28.408959 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7349c8f9-ad1c-4422-8003-73e3419ad9c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:28.409222 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7349c8f9-ad1c-4422-8003-73e3419ad9c0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:28.477844 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7349c8f9-ad1c-4422-8003-73e3419ad9c0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:28.479276 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7349c8f9-ad1c-4422-8003-73e3419ad9c0 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:28.530697 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:28.590965 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 557/1104] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:43:28 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 25 00:43:28.597690 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e130639-cd62-460e-a22b-4bb0feee374c None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:28.626762 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:28.713339 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e130639-cd62-460e-a22b-4bb0feee374c None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:28.748471 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbbe6736-abd6-434b-88b7-9f2dc3b4c1f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:28.749088 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbbe6736-abd6-434b-88b7-9f2dc3b4c1f0 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:28.749398 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbbe6736-abd6-434b-88b7-9f2dc3b4c1f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:28.749664 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbbe6736-abd6-434b-88b7-9f2dc3b4c1f0 None admin] PATH_INFO: `/v3/projects/36a6c7219a784408a5462f30fa36d521` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:28.750747 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbbe6736-abd6-434b-88b7-9f2dc3b4c1f0 None admin] RBAC: Authorizing `identity:get_project(project_id=36a6c7219a784408a5462f30fa36d521)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:28.752149 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbbe6736-abd6-434b-88b7-9f2dc3b4c1f0 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:28.753789 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 548/1105] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:43:28 2022] GET /identity/v3/projects/36a6c7219a784408a5462f30fa36d521 => generated 311 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:28.760034 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7349c8f9-ad1c-4422-8003-73e3419ad9c0 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:28.857715 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7349c8f9-ad1c-4422-8003-73e3419ad9c0 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:28.891815 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38efedb2-0643-49ad-9fae-d484a0cbc4e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:28.892412 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38efedb2-0643-49ad-9fae-d484a0cbc4e4 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:28.892684 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38efedb2-0643-49ad-9fae-d484a0cbc4e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:28.892949 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38efedb2-0643-49ad-9fae-d484a0cbc4e4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:28.893690 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38efedb2-0643-49ad-9fae-d484a0cbc4e4 None admin] RBAC: Adding query filter params (role.id=2c242abf6ed8410ab0a4a354ca1794fb, scope.project.id=36a6c7219a784408a5462f30fa36d521, user.id=17e2a2075cb44c5c98ea296a226f103a) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:28.893992 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38efedb2-0643-49ad-9fae-d484a0cbc4e4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:28.895946 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38efedb2-0643-49ad-9fae-d484a0cbc4e4 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:28.901866 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 558/1106] 10.222.0.52 () {56 vars in 1374 bytes} [Sun Sep 25 00:43:28 2022] GET /identity/v3/role_assignments?user.id=17e2a2075cb44c5c98ea296a226f103a&scope.project.id=36a6c7219a784408a5462f30fa36d521&role.id=2c242abf6ed8410ab0a4a354ca1794fb => generated 608 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:31.105978 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37270aa-143d-4ff1-9da8-ea3ae9ab8dc8 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:31.106522 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37270aa-143d-4ff1-9da8-ea3ae9ab8dc8 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:31.106522 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37270aa-143d-4ff1-9da8-ea3ae9ab8dc8 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:31.107825 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 549/1107] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:43:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:43:31.113503 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367153e3-8576-4760-a834-cdec5e01f0b3 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:31.113977 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367153e3-8576-4760-a834-cdec5e01f0b3 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:31.114198 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367153e3-8576-4760-a834-cdec5e01f0b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:31.133870 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-367153e3-8576-4760-a834-cdec5e01f0b3 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:43:31.135049 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-367153e3-8576-4760-a834-cdec5e01f0b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:43:31.146822 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:31.204798 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 559/1108] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:43:31 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:43:31.483873 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e37270aa-143d-4ff1-9da8-ea3ae9ab8dc8 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:31.583339 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:31.593369 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e37270aa-143d-4ff1-9da8-ea3ae9ab8dc8 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:31.626976 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e241f1f3-ee89-48d7-a3cb-688a5b9cfe62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:31.627586 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e241f1f3-ee89-48d7-a3cb-688a5b9cfe62 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:31.627859 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e241f1f3-ee89-48d7-a3cb-688a5b9cfe62 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:31.628121 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e241f1f3-ee89-48d7-a3cb-688a5b9cfe62 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:31.629848 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e241f1f3-ee89-48d7-a3cb-688a5b9cfe62 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:31.631258 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e241f1f3-ee89-48d7-a3cb-688a5b9cfe62 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:31.632805 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 550/1109] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:43:31 2022] GET /identity/v3/domains/default => generated 199 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:31.638504 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-367153e3-8576-4760-a834-cdec5e01f0b3 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:31.733283 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-367153e3-8576-4760-a834-cdec5e01f0b3 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:31.766103 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a5baf8-2acb-474b-bb98-9e2663fa74f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:31.766686 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a5baf8-2acb-474b-bb98-9e2663fa74f4 None admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:31.766956 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a5baf8-2acb-474b-bb98-9e2663fa74f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:31.767213 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a5baf8-2acb-474b-bb98-9e2663fa74f4 None admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:31.767850 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25a5baf8-2acb-474b-bb98-9e2663fa74f4 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:31.769246 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25a5baf8-2acb-474b-bb98-9e2663fa74f4 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:31.786581 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:31.791927 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 560/1110] 10.222.0.52 () {60 vars in 1132 bytes} [Sun Sep 25 00:43:31 2022] POST /identity/v3/users => generated 286 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:43:34.025731 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a15578-f82f-42df-a8c0-6e960df5b6b3 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:34.026326 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a15578-f82f-42df-a8c0-6e960df5b6b3 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:34.026326 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a15578-f82f-42df-a8c0-6e960df5b6b3 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:34.027605 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 551/1111] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:43:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:43:34.033386 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18da722-dc30-4d7d-aed7-5b0072f71fca None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:34.033875 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18da722-dc30-4d7d-aed7-5b0072f71fca None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:34.033975 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18da722-dc30-4d7d-aed7-5b0072f71fca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:34.060628 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-e18da722-dc30-4d7d-aed7-5b0072f71fca None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:43:34.062887 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-e18da722-dc30-4d7d-aed7-5b0072f71fca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:43:34.132899 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 561/1112] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:43:34 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:43:34.402730 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a15578-f82f-42df-a8c0-6e960df5b6b3 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:34.505303 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a15578-f82f-42df-a8c0-6e960df5b6b3 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:34.541833 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7192a036-4618-485e-adbd-4b6bd58401db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:34.542462 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7192a036-4618-485e-adbd-4b6bd58401db None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:34.542738 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7192a036-4618-485e-adbd-4b6bd58401db None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:34.543008 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7192a036-4618-485e-adbd-4b6bd58401db None admin] PATH_INFO: `/v3/roles/a917909d7aa84d00a841c77e7e7bdc8a` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:34.544408 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7192a036-4618-485e-adbd-4b6bd58401db None admin] RBAC: Authorizing `identity:get_role(role_id=a917909d7aa84d00a841c77e7e7bdc8a)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:34.545793 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7192a036-4618-485e-adbd-4b6bd58401db None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:34.547174 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 552/1113] 10.222.0.52 () {56 vars in 1177 bytes} [Sun Sep 25 00:43:34 2022] GET /identity/v3/roles/a917909d7aa84d00a841c77e7e7bdc8a => generated 223 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:34.553288 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e18da722-dc30-4d7d-aed7-5b0072f71fca None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:34.651534 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e18da722-dc30-4d7d-aed7-5b0072f71fca None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:34.679115 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:34.684856 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ba0ebe-1faf-481e-89fa-b8c4f3a68cfa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:34.685496 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ba0ebe-1faf-481e-89fa-b8c4f3a68cfa None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:34.685769 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ba0ebe-1faf-481e-89fa-b8c4f3a68cfa None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:34.686029 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ba0ebe-1faf-481e-89fa-b8c4f3a68cfa None admin] PATH_INFO: `/v3/users/159de4ea4ea64e05b64f68b07316db6b` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:34.699454 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5ba0ebe-1faf-481e-89fa-b8c4f3a68cfa None admin] RBAC: Authorizing `identity:get_user(user_id=159de4ea4ea64e05b64f68b07316db6b)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:34.700864 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5ba0ebe-1faf-481e-89fa-b8c4f3a68cfa None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:34.702490 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 562/1114] 10.222.0.52 () {56 vars in 1177 bytes} [Sun Sep 25 00:43:34 2022] GET /identity/v3/users/159de4ea4ea64e05b64f68b07316db6b => generated 286 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:34.708762 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7192a036-4618-485e-adbd-4b6bd58401db None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:34.736862 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:34.809184 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7192a036-4618-485e-adbd-4b6bd58401db None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:34.845124 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e611a4-1f4b-45db-99e5-1218d45a6688 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:34.845754 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e611a4-1f4b-45db-99e5-1218d45a6688 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:34.846028 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e611a4-1f4b-45db-99e5-1218d45a6688 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:34.846293 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e611a4-1f4b-45db-99e5-1218d45a6688 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:34.914766 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5e611a4-1f4b-45db-99e5-1218d45a6688 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:34.916197 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5e611a4-1f4b-45db-99e5-1218d45a6688 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:34.943375 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:35.016362 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 553/1115] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:43:34 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 25 00:43:35.022835 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ba0ebe-1faf-481e-89fa-b8c4f3a68cfa None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:35.119383 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ba0ebe-1faf-481e-89fa-b8c4f3a68cfa None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:35.153839 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-308713d8-ebb4-4551-b533-9590d4b173dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:35.154460 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308713d8-ebb4-4551-b533-9590d4b173dd None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:35.154736 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308713d8-ebb4-4551-b533-9590d4b173dd None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:35.155003 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308713d8-ebb4-4551-b533-9590d4b173dd None admin] PATH_INFO: `/v3/projects/36a6c7219a784408a5462f30fa36d521` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:35.156107 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-308713d8-ebb4-4551-b533-9590d4b173dd None admin] RBAC: Authorizing `identity:get_project(project_id=36a6c7219a784408a5462f30fa36d521)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:35.157548 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-308713d8-ebb4-4551-b533-9590d4b173dd None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:35.159182 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 563/1116] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:43:35 2022] GET /identity/v3/projects/36a6c7219a784408a5462f30fa36d521 => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:35.165600 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e611a4-1f4b-45db-99e5-1218d45a6688 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:35.264217 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e611a4-1f4b-45db-99e5-1218d45a6688 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:35.290992 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:35.299115 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e135cd6-adc3-4151-bd37-7364687f6d4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:35.299712 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e135cd6-adc3-4151-bd37-7364687f6d4e None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:35.299981 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e135cd6-adc3-4151-bd37-7364687f6d4e None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:35.300243 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e135cd6-adc3-4151-bd37-7364687f6d4e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:35.301619 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e135cd6-adc3-4151-bd37-7364687f6d4e None admin] RBAC: Adding query filter params (role.id=a917909d7aa84d00a841c77e7e7bdc8a, scope.project.id=36a6c7219a784408a5462f30fa36d521, user.id=159de4ea4ea64e05b64f68b07316db6b) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:35.302155 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e135cd6-adc3-4151-bd37-7364687f6d4e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:35.303691 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e135cd6-adc3-4151-bd37-7364687f6d4e None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:35.309138 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 554/1117] 10.222.0.52 () {56 vars in 1374 bytes} [Sun Sep 25 00:43:35 2022] GET /identity/v3/role_assignments?user.id=159de4ea4ea64e05b64f68b07316db6b&scope.project.id=36a6c7219a784408a5462f30fa36d521&role.id=a917909d7aa84d00a841c77e7e7bdc8a => generated 259 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:37.440756 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d692dff-0377-42e1-8ff8-3d98b7be6cc6 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:37.441905 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d692dff-0377-42e1-8ff8-3d98b7be6cc6 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:37.441905 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d692dff-0377-42e1-8ff8-3d98b7be6cc6 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:37.443162 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 564/1118] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:43:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:43:37.448365 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5743bb-9512-4058-a68d-8af5353f9d99 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:37.448777 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5743bb-9512-4058-a68d-8af5353f9d99 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:37.448886 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5743bb-9512-4058-a68d-8af5353f9d99 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:37.468994 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-cb5743bb-9512-4058-a68d-8af5353f9d99 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:43:37.470280 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-cb5743bb-9512-4058-a68d-8af5353f9d99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:43:37.539985 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 555/1119] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:43:37 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:43:37.819000 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d692dff-0377-42e1-8ff8-3d98b7be6cc6 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:37.874908 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:37.919791 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d692dff-0377-42e1-8ff8-3d98b7be6cc6 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:37.953895 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a79b4be-26b1-4dfd-9a78-b6412c8a795f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:37.954516 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a79b4be-26b1-4dfd-9a78-b6412c8a795f None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:37.954788 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a79b4be-26b1-4dfd-9a78-b6412c8a795f None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:37.955056 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a79b4be-26b1-4dfd-9a78-b6412c8a795f None admin] PATH_INFO: `/v3/roles/a917909d7aa84d00a841c77e7e7bdc8a` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:37.956434 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a79b4be-26b1-4dfd-9a78-b6412c8a795f None admin] RBAC: Authorizing `identity:get_role(role_id=a917909d7aa84d00a841c77e7e7bdc8a)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:37.957995 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a79b4be-26b1-4dfd-9a78-b6412c8a795f None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:37.959325 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 565/1120] 10.222.0.52 () {56 vars in 1177 bytes} [Sun Sep 25 00:43:37 2022] GET /identity/v3/roles/a917909d7aa84d00a841c77e7e7bdc8a => generated 223 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:37.965622 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5743bb-9512-4058-a68d-8af5353f9d99 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:38.069736 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5743bb-9512-4058-a68d-8af5353f9d99 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:38.093787 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:38.108308 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e070385-70dc-44fb-91df-8292d1336a90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:38.108941 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e070385-70dc-44fb-91df-8292d1336a90 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:38.109216 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e070385-70dc-44fb-91df-8292d1336a90 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:38.109516 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e070385-70dc-44fb-91df-8292d1336a90 None admin] PATH_INFO: `/v3/users/159de4ea4ea64e05b64f68b07316db6b` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:38.110642 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e070385-70dc-44fb-91df-8292d1336a90 None admin] RBAC: Authorizing `identity:get_user(user_id=159de4ea4ea64e05b64f68b07316db6b)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:38.112065 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e070385-70dc-44fb-91df-8292d1336a90 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:38.114932 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 556/1121] 10.222.0.52 () {56 vars in 1177 bytes} [Sun Sep 25 00:43:37 2022] GET /identity/v3/users/159de4ea4ea64e05b64f68b07316db6b => generated 286 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:38.120761 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a79b4be-26b1-4dfd-9a78-b6412c8a795f None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:38.225371 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a79b4be-26b1-4dfd-9a78-b6412c8a795f None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:38.251332 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:38.264507 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85b4e855-7dc6-4d64-adbe-7a28d8d08c01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:38.265148 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b4e855-7dc6-4d64-adbe-7a28d8d08c01 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:38.265456 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b4e855-7dc6-4d64-adbe-7a28d8d08c01 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:38.265726 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b4e855-7dc6-4d64-adbe-7a28d8d08c01 None admin] PATH_INFO: `/v3/projects/36a6c7219a784408a5462f30fa36d521` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:38.266882 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85b4e855-7dc6-4d64-adbe-7a28d8d08c01 None admin] RBAC: Authorizing `identity:get_project(project_id=36a6c7219a784408a5462f30fa36d521)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:38.268330 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85b4e855-7dc6-4d64-adbe-7a28d8d08c01 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:38.271450 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 566/1122] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:43:38 2022] GET /identity/v3/projects/36a6c7219a784408a5462f30fa36d521 => generated 311 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:38.277599 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e070385-70dc-44fb-91df-8292d1336a90 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:38.385922 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e070385-70dc-44fb-91df-8292d1336a90 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:38.423644 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53fc1ac-637a-4012-8ac6-ce76a4d5d53c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:38.424300 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53fc1ac-637a-4012-8ac6-ce76a4d5d53c None admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:38.424571 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53fc1ac-637a-4012-8ac6-ce76a4d5d53c None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:38.424835 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53fc1ac-637a-4012-8ac6-ce76a4d5d53c None admin] PATH_INFO: `/v3/projects/36a6c7219a784408a5462f30fa36d521/users/159de4ea4ea64e05b64f68b07316db6b/roles/a917909d7aa84d00a841c77e7e7bdc8a` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:38.427236 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d53fc1ac-637a-4012-8ac6-ce76a4d5d53c None admin] RBAC: Authorizing `identity:create_grant(project_id=36a6c7219a784408a5462f30fa36d521, user_id=159de4ea4ea64e05b64f68b07316db6b, role_id=a917909d7aa84d00a841c77e7e7bdc8a)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:38.428658 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d53fc1ac-637a-4012-8ac6-ce76a4d5d53c None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:38.443824 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 557/1123] 10.222.0.52 () {58 vars in 1439 bytes} [Sun Sep 25 00:43:38 2022] PUT /identity/v3/projects/36a6c7219a784408a5462f30fa36d521/users/159de4ea4ea64e05b64f68b07316db6b/roles/a917909d7aa84d00a841c77e7e7bdc8a => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:43:40.588732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e4361b-294a-44b1-8e1d-8ef321b3b710 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:40.589315 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e4361b-294a-44b1-8e1d-8ef321b3b710 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:40.589315 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e4361b-294a-44b1-8e1d-8ef321b3b710 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:40.590625 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 567/1124] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:43:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:43:40.596449 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d783c3e3-a6f1-40c7-b93c-25bd3dc56a4d None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:40.596969 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d783c3e3-a6f1-40c7-b93c-25bd3dc56a4d None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:40.596969 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d783c3e3-a6f1-40c7-b93c-25bd3dc56a4d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:40.615995 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-d783c3e3-a6f1-40c7-b93c-25bd3dc56a4d None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:43:40.617170 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-d783c3e3-a6f1-40c7-b93c-25bd3dc56a4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:43:40.663889 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:40.686097 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 558/1125] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:43:40 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:43:40.960949 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e4361b-294a-44b1-8e1d-8ef321b3b710 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:41.064375 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e4361b-294a-44b1-8e1d-8ef321b3b710 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:41.099239 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8317906-bbe0-4c77-89a8-9297b3e7c365 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:41.099855 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8317906-bbe0-4c77-89a8-9297b3e7c365 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:41.100131 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8317906-bbe0-4c77-89a8-9297b3e7c365 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:41.100404 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8317906-bbe0-4c77-89a8-9297b3e7c365 None admin] PATH_INFO: `/v3/roles/a917909d7aa84d00a841c77e7e7bdc8a` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:41.101829 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8317906-bbe0-4c77-89a8-9297b3e7c365 None admin] RBAC: Authorizing `identity:get_role(role_id=a917909d7aa84d00a841c77e7e7bdc8a)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:41.103183 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8317906-bbe0-4c77-89a8-9297b3e7c365 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:41.104508 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 568/1126] 10.222.0.52 () {56 vars in 1177 bytes} [Sun Sep 25 00:43:40 2022] GET /identity/v3/roles/a917909d7aa84d00a841c77e7e7bdc8a => generated 223 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:41.110439 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d783c3e3-a6f1-40c7-b93c-25bd3dc56a4d None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:41.210662 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d783c3e3-a6f1-40c7-b93c-25bd3dc56a4d None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:41.246987 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad09c35-966c-45af-9b14-74c6380004d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:41.247602 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad09c35-966c-45af-9b14-74c6380004d6 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:41.247871 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad09c35-966c-45af-9b14-74c6380004d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:41.248132 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad09c35-966c-45af-9b14-74c6380004d6 None admin] PATH_INFO: `/v3/users/159de4ea4ea64e05b64f68b07316db6b` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:41.249227 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fad09c35-966c-45af-9b14-74c6380004d6 None admin] RBAC: Authorizing `identity:get_user(user_id=159de4ea4ea64e05b64f68b07316db6b)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:41.250670 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fad09c35-966c-45af-9b14-74c6380004d6 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:41.252243 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 559/1127] 10.222.0.52 () {56 vars in 1177 bytes} [Sun Sep 25 00:43:41 2022] GET /identity/v3/users/159de4ea4ea64e05b64f68b07316db6b => generated 286 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:41.258756 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8317906-bbe0-4c77-89a8-9297b3e7c365 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:41.317916 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:41.358662 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8317906-bbe0-4c77-89a8-9297b3e7c365 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:41.391648 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caac52a5-fa0a-4901-9356-6d81b613528d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:41.392245 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caac52a5-fa0a-4901-9356-6d81b613528d None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:41.392517 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caac52a5-fa0a-4901-9356-6d81b613528d None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:41.392783 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caac52a5-fa0a-4901-9356-6d81b613528d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:41.458263 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caac52a5-fa0a-4901-9356-6d81b613528d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:41.459679 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caac52a5-fa0a-4901-9356-6d81b613528d None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:41.516904 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:41.557980 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 569/1128] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:43:41 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 300 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 25 00:43:41.563914 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad09c35-966c-45af-9b14-74c6380004d6 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:41.613878 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:41.666269 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad09c35-966c-45af-9b14-74c6380004d6 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:41.701739 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edbb5589-c279-4f11-8641-055306f90296 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:41.702354 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edbb5589-c279-4f11-8641-055306f90296 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:41.702625 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edbb5589-c279-4f11-8641-055306f90296 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:41.702889 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edbb5589-c279-4f11-8641-055306f90296 None admin] PATH_INFO: `/v3/projects/36a6c7219a784408a5462f30fa36d521` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:41.703980 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edbb5589-c279-4f11-8641-055306f90296 None admin] RBAC: Authorizing `identity:get_project(project_id=36a6c7219a784408a5462f30fa36d521)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:41.705694 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edbb5589-c279-4f11-8641-055306f90296 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:41.707738 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 560/1129] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:43:41 2022] GET /identity/v3/projects/36a6c7219a784408a5462f30fa36d521 => generated 311 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:41.713929 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caac52a5-fa0a-4901-9356-6d81b613528d None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:41.811315 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caac52a5-fa0a-4901-9356-6d81b613528d None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:41.844415 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96b1bb8-3d0b-416c-a5b6-7cbd7d563b8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:41.844997 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96b1bb8-3d0b-416c-a5b6-7cbd7d563b8e None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:41.845285 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96b1bb8-3d0b-416c-a5b6-7cbd7d563b8e None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:41.845559 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96b1bb8-3d0b-416c-a5b6-7cbd7d563b8e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:41.846265 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a96b1bb8-3d0b-416c-a5b6-7cbd7d563b8e None admin] RBAC: Adding query filter params (role.id=a917909d7aa84d00a841c77e7e7bdc8a, scope.project.id=36a6c7219a784408a5462f30fa36d521, user.id=159de4ea4ea64e05b64f68b07316db6b) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:41.846562 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a96b1bb8-3d0b-416c-a5b6-7cbd7d563b8e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:41.848479 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a96b1bb8-3d0b-416c-a5b6-7cbd7d563b8e None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:41.854116 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 570/1130] 10.222.0.52 () {56 vars in 1374 bytes} [Sun Sep 25 00:43:41 2022] GET /identity/v3/role_assignments?user.id=159de4ea4ea64e05b64f68b07316db6b&scope.project.id=36a6c7219a784408a5462f30fa36d521&role.id=a917909d7aa84d00a841c77e7e7bdc8a => generated 608 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:44.068053 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3d81db-3381-4050-aad4-a156d85b517b None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:44.068594 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3d81db-3381-4050-aad4-a156d85b517b None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:44.068594 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3d81db-3381-4050-aad4-a156d85b517b None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:44.069918 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 561/1131] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:43:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:43:44.075232 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a7d912a-ccd5-4c79-9260-c79c4e74018e None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:44.075630 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a7d912a-ccd5-4c79-9260-c79c4e74018e None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:44.075771 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a7d912a-ccd5-4c79-9260-c79c4e74018e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:44.094555 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-6a7d912a-ccd5-4c79-9260-c79c4e74018e None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:43:44.095735 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-6a7d912a-ccd5-4c79-9260-c79c4e74018e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:43:44.114370 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:44.165093 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 571/1132] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:43:44 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:43:44.432090 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae3d81db-3381-4050-aad4-a156d85b517b None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:44.537564 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae3d81db-3381-4050-aad4-a156d85b517b None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:44.556749 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:44.576570 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f008d146-f0fa-4738-981c-b0695cf63c85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:44.577206 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f008d146-f0fa-4738-981c-b0695cf63c85 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:44.578272 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f008d146-f0fa-4738-981c-b0695cf63c85 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:44.578659 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f008d146-f0fa-4738-981c-b0695cf63c85 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:44.580602 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f008d146-f0fa-4738-981c-b0695cf63c85 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:44.583427 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f008d146-f0fa-4738-981c-b0695cf63c85 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:44.586562 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 562/1133] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:43:44 2022] GET /identity/v3/domains/default => generated 199 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:44.592835 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a7d912a-ccd5-4c79-9260-c79c4e74018e None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:44.699544 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a7d912a-ccd5-4c79-9260-c79c4e74018e None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:44.733206 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e9f25fa-fcf5-418d-a4dd-b4dfaf8679ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:44.733827 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e9f25fa-fcf5-418d-a4dd-b4dfaf8679ed None admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:44.734095 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e9f25fa-fcf5-418d-a4dd-b4dfaf8679ed None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:44.734357 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e9f25fa-fcf5-418d-a4dd-b4dfaf8679ed None admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:44.734993 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e9f25fa-fcf5-418d-a4dd-b4dfaf8679ed None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:44.736396 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e9f25fa-fcf5-418d-a4dd-b4dfaf8679ed None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:44.768882 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 572/1134] 10.222.0.52 () {60 vars in 1141 bytes} [Sun Sep 25 00:43:44 2022] POST /identity/v3/projects => generated 311 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:43:46.994435 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe787272-1f9c-47cc-bd7b-da556d772c4e None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:46.995033 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe787272-1f9c-47cc-bd7b-da556d772c4e None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:46.995033 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe787272-1f9c-47cc-bd7b-da556d772c4e None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:46.996352 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 563/1135] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:43:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:43:47.001835 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e297a90-dcec-4417-baca-03342350f738 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:47.002302 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e297a90-dcec-4417-baca-03342350f738 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:47.002393 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e297a90-dcec-4417-baca-03342350f738 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:47.016002 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:47.022475 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-0e297a90-dcec-4417-baca-03342350f738 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:43:47.024193 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-0e297a90-dcec-4417-baca-03342350f738 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:43:47.120588 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 573/1136] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:43:47 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:43:47.392218 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe787272-1f9c-47cc-bd7b-da556d772c4e None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:47.502301 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe787272-1f9c-47cc-bd7b-da556d772c4e None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:47.537381 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03ba805b-cbf9-441d-95ce-e753a0b88f04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:47.538007 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ba805b-cbf9-441d-95ce-e753a0b88f04 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:47.538282 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ba805b-cbf9-441d-95ce-e753a0b88f04 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:47.538548 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ba805b-cbf9-441d-95ce-e753a0b88f04 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:47.539699 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03ba805b-cbf9-441d-95ce-e753a0b88f04 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:47.541125 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03ba805b-cbf9-441d-95ce-e753a0b88f04 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:47.542755 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 564/1137] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:43:47 2022] GET /identity/v3/domains/default => generated 199 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:47.548808 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e297a90-dcec-4417-baca-03342350f738 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:47.654124 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e297a90-dcec-4417-baca-03342350f738 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:47.696185 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea2d07b-3907-4999-9c7e-00fcb80a5b2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:47.696784 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea2d07b-3907-4999-9c7e-00fcb80a5b2c None admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:47.697057 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea2d07b-3907-4999-9c7e-00fcb80a5b2c None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:47.697346 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea2d07b-3907-4999-9c7e-00fcb80a5b2c None admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:47.697994 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cea2d07b-3907-4999-9c7e-00fcb80a5b2c None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:47.699410 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cea2d07b-3907-4999-9c7e-00fcb80a5b2c None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:47.709321 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:47.727641 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 574/1138] 10.222.0.52 () {60 vars in 1132 bytes} [Sun Sep 25 00:43:47 2022] POST /identity/v3/users => generated 286 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:43:50.003706 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9796d62e-88d5-4e93-b351-4c5d222e6952 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:50.004319 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9796d62e-88d5-4e93-b351-4c5d222e6952 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:50.004319 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9796d62e-88d5-4e93-b351-4c5d222e6952 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:50.005983 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 565/1139] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:43:50 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 00:43:50.011986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa6bd1a-1a7e-4ce4-b1e0-64759d1eb8d6 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:50.012533 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa6bd1a-1a7e-4ce4-b1e0-64759d1eb8d6 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:50.012533 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa6bd1a-1a7e-4ce4-b1e0-64759d1eb8d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:50.032361 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-7fa6bd1a-1a7e-4ce4-b1e0-64759d1eb8d6 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:43:50.033618 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-7fa6bd1a-1a7e-4ce4-b1e0-64759d1eb8d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:43:50.105874 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 575/1140] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:43:50 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:43:50.377613 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9796d62e-88d5-4e93-b351-4c5d222e6952 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:50.437283 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:50.495110 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9796d62e-88d5-4e93-b351-4c5d222e6952 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:50.531874 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2ec309-c465-4d88-a386-8b91003309c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:50.532557 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2ec309-c465-4d88-a386-8b91003309c4 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:50.532875 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2ec309-c465-4d88-a386-8b91003309c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:50.533186 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2ec309-c465-4d88-a386-8b91003309c4 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:50.544580 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c2ec309-c465-4d88-a386-8b91003309c4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:43:50.549544 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-2c2ec309-c465-4d88-a386-8b91003309c4 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:43:50.549544 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:50.549544 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:43:50.549544 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:43:50.549544 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:43:50.549544 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:43:50.549544 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:43:50.549544 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:43:50.549544 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:43:50.549544 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:43:50.549544 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:43:50.549544 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:43:50.549544 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:43:50.549544 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:43:50.549544 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:43:50.549544 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:43:50.551829 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:43:50.551829 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:43:50.551829 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:43:50.551829 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:43:50.551829 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:43:50.551829 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:43:50.551829 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:43:50.551829 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:43:50.551829 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:43:50.551829 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:43:50.551829 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:43:50.551829 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:43:50.551829 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:43:50.551829 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:43:50.551829 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:43:50.551829 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:43:50.553585 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:43:50.553585 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:43:50.553585 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:43:50.553585 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:43:50.553585 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:43:50.553585 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:43:50.553585 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:43:50.553585 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:43:50.553585 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:43:50.553585 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:43:50.553585 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:43:50.553585 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:43:50.553585 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 566/1141] 10.222.0.52 () {56 vars in 1096 bytes} [Sun Sep 25 00:43:50 2022] GET /identity/v3/roles/admin => generated 83 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:43:50.559200 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa6bd1a-1a7e-4ce4-b1e0-64759d1eb8d6 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:50.674062 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa6bd1a-1a7e-4ce4-b1e0-64759d1eb8d6 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:50.701227 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:50.715709 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c840633-11ee-4b5d-893c-d79436ff952f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:50.716395 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c840633-11ee-4b5d-893c-d79436ff952f None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:50.716715 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c840633-11ee-4b5d-893c-d79436ff952f None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:50.717026 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c840633-11ee-4b5d-893c-d79436ff952f None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:50.717780 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c840633-11ee-4b5d-893c-d79436ff952f None admin] RBAC: Adding query filter params (name=admin) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:50.718122 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c840633-11ee-4b5d-893c-d79436ff952f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:50.719583 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c840633-11ee-4b5d-893c-d79436ff952f None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:50.729091 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 576/1142] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:43:50 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:50.735324 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2ec309-c465-4d88-a386-8b91003309c4 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:50.842437 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2ec309-c465-4d88-a386-8b91003309c4 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:50.857459 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:50.881317 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f94ac072-280b-4921-abc8-71cb100b8d05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:50.881962 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94ac072-280b-4921-abc8-71cb100b8d05 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:50.882239 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94ac072-280b-4921-abc8-71cb100b8d05 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:50.882511 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94ac072-280b-4921-abc8-71cb100b8d05 None admin] PATH_INFO: `/v3/users/81dce634acf24fcaa7295f08ec1b87dc` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:50.898315 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f94ac072-280b-4921-abc8-71cb100b8d05 None admin] RBAC: Authorizing `identity:get_user(user_id=81dce634acf24fcaa7295f08ec1b87dc)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:50.899767 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f94ac072-280b-4921-abc8-71cb100b8d05 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:50.901501 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 567/1143] 10.222.0.52 () {56 vars in 1177 bytes} [Sun Sep 25 00:43:50 2022] GET /identity/v3/users/81dce634acf24fcaa7295f08ec1b87dc => generated 286 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:50.907605 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c840633-11ee-4b5d-893c-d79436ff952f None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:51.013894 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c840633-11ee-4b5d-893c-d79436ff952f None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:51.048842 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf639dc-22a2-4713-9ca9-79b4d2852d2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:51.049469 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf639dc-22a2-4713-9ca9-79b4d2852d2d None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:51.049742 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf639dc-22a2-4713-9ca9-79b4d2852d2d None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:51.050611 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf639dc-22a2-4713-9ca9-79b4d2852d2d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:51.105704 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:51.122922 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bf639dc-22a2-4713-9ca9-79b4d2852d2d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:51.124366 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bf639dc-22a2-4713-9ca9-79b4d2852d2d None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:51.254037 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 577/1144] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:43:50 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 25 00:43:51.263445 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f94ac072-280b-4921-abc8-71cb100b8d05 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:51.371813 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f94ac072-280b-4921-abc8-71cb100b8d05 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:51.409003 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d435460-6cf3-44c0-b20e-dd4c0b113721 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:51.409763 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d435460-6cf3-44c0-b20e-dd4c0b113721 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:51.410039 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d435460-6cf3-44c0-b20e-dd4c0b113721 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:51.410303 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d435460-6cf3-44c0-b20e-dd4c0b113721 None admin] PATH_INFO: `/v3/projects/1e920620636443fb91e0138f970fd6e4` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:51.411439 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d435460-6cf3-44c0-b20e-dd4c0b113721 None admin] RBAC: Authorizing `identity:get_project(project_id=1e920620636443fb91e0138f970fd6e4)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:51.412853 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d435460-6cf3-44c0-b20e-dd4c0b113721 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:51.414539 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 568/1145] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:43:51 2022] GET /identity/v3/projects/1e920620636443fb91e0138f970fd6e4 => generated 311 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:51.424295 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf639dc-22a2-4713-9ca9-79b4d2852d2d None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:51.526196 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:51.553829 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf639dc-22a2-4713-9ca9-79b4d2852d2d None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:51.611718 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-322341c8-6574-458f-bb27-60e2938136b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:51.613033 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322341c8-6574-458f-bb27-60e2938136b7 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:51.613550 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322341c8-6574-458f-bb27-60e2938136b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:51.614261 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322341c8-6574-458f-bb27-60e2938136b7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:51.615602 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-322341c8-6574-458f-bb27-60e2938136b7 None admin] RBAC: Adding query filter params (role.id=2c242abf6ed8410ab0a4a354ca1794fb, scope.project.id=1e920620636443fb91e0138f970fd6e4, user.id=81dce634acf24fcaa7295f08ec1b87dc) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:51.616142 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-322341c8-6574-458f-bb27-60e2938136b7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:51.618795 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-322341c8-6574-458f-bb27-60e2938136b7 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:51.629464 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 578/1146] 10.222.0.52 () {56 vars in 1374 bytes} [Sun Sep 25 00:43:51 2022] GET /identity/v3/role_assignments?user.id=81dce634acf24fcaa7295f08ec1b87dc&scope.project.id=1e920620636443fb91e0138f970fd6e4&role.id=2c242abf6ed8410ab0a4a354ca1794fb => generated 259 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:53.860804 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d84ea2-0be1-402e-8919-e0a6a5a09105 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:53.861325 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d84ea2-0be1-402e-8919-e0a6a5a09105 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:53.861325 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d84ea2-0be1-402e-8919-e0a6a5a09105 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:53.862880 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 569/1147] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:43:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:43:53.868712 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813ae2e4-3479-418c-a220-0abb66c11177 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:53.869137 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813ae2e4-3479-418c-a220-0abb66c11177 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:53.869250 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813ae2e4-3479-418c-a220-0abb66c11177 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:53.887541 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-813ae2e4-3479-418c-a220-0abb66c11177 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:43:53.888729 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-813ae2e4-3479-418c-a220-0abb66c11177 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:43:53.955621 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 579/1148] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:43:53 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:43:54.224841 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d84ea2-0be1-402e-8919-e0a6a5a09105 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:54.266278 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:54.326446 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d84ea2-0be1-402e-8919-e0a6a5a09105 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:54.360350 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75df788b-9862-42c1-9346-4ae1453498c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:54.360961 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75df788b-9862-42c1-9346-4ae1453498c8 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:54.361231 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75df788b-9862-42c1-9346-4ae1453498c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:54.361523 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75df788b-9862-42c1-9346-4ae1453498c8 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:54.374239 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75df788b-9862-42c1-9346-4ae1453498c8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:43:54.379131 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-75df788b-9862-42c1-9346-4ae1453498c8 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:43:54.379131 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:54.379131 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:43:54.379131 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:43:54.379131 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:43:54.379131 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:43:54.379131 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:43:54.379131 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:43:54.379131 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:43:54.379131 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:43:54.379131 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:43:54.379131 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:43:54.379131 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:43:54.379131 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:43:54.379131 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:43:54.379131 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:43:54.380245 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:43:54.380245 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:43:54.380245 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:43:54.380245 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:43:54.380245 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:43:54.380245 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:43:54.380245 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:43:54.380245 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:43:54.380245 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:43:54.380245 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:43:54.380245 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:43:54.380245 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:43:54.380245 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:43:54.380245 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:43:54.380245 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:43:54.380245 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:43:54.381284 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:43:54.381284 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:43:54.381284 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:43:54.381284 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:43:54.381284 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:43:54.381284 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:43:54.381284 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:43:54.381284 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:43:54.381284 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:43:54.381284 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:43:54.381284 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:43:54.381284 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:43:54.381284 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 570/1149] 10.222.0.52 () {56 vars in 1096 bytes} [Sun Sep 25 00:43:54 2022] GET /identity/v3/roles/admin => generated 83 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:43:54.386328 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-813ae2e4-3479-418c-a220-0abb66c11177 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:54.464185 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:54.487724 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-813ae2e4-3479-418c-a220-0abb66c11177 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:54.524659 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12949249-13ea-4c2c-9bdc-22c7691d9788 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:54.525277 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12949249-13ea-4c2c-9bdc-22c7691d9788 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:54.525563 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12949249-13ea-4c2c-9bdc-22c7691d9788 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:54.525832 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12949249-13ea-4c2c-9bdc-22c7691d9788 None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:54.526491 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12949249-13ea-4c2c-9bdc-22c7691d9788 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:54.526791 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12949249-13ea-4c2c-9bdc-22c7691d9788 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:54.528152 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12949249-13ea-4c2c-9bdc-22c7691d9788 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:54.536141 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 580/1150] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:43:54 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:54.542079 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75df788b-9862-42c1-9346-4ae1453498c8 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:54.638325 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:54.642233 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75df788b-9862-42c1-9346-4ae1453498c8 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:54.678780 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d2b9dbb-0c5c-4c5e-b60d-3a06ccf85147 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:54.679553 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2b9dbb-0c5c-4c5e-b60d-3a06ccf85147 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:54.679862 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2b9dbb-0c5c-4c5e-b60d-3a06ccf85147 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:54.680266 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2b9dbb-0c5c-4c5e-b60d-3a06ccf85147 None admin] PATH_INFO: `/v3/users/81dce634acf24fcaa7295f08ec1b87dc` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:54.681433 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d2b9dbb-0c5c-4c5e-b60d-3a06ccf85147 None admin] RBAC: Authorizing `identity:get_user(user_id=81dce634acf24fcaa7295f08ec1b87dc)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:54.682864 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d2b9dbb-0c5c-4c5e-b60d-3a06ccf85147 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:54.684447 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 571/1151] 10.222.0.52 () {56 vars in 1177 bytes} [Sun Sep 25 00:43:54 2022] GET /identity/v3/users/81dce634acf24fcaa7295f08ec1b87dc => generated 286 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:54.690300 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12949249-13ea-4c2c-9bdc-22c7691d9788 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:54.790035 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12949249-13ea-4c2c-9bdc-22c7691d9788 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:54.822890 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d641469-04a2-492f-ab7a-faf28fad9433 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:54.823505 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d641469-04a2-492f-ab7a-faf28fad9433 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:54.823774 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d641469-04a2-492f-ab7a-faf28fad9433 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:54.824035 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d641469-04a2-492f-ab7a-faf28fad9433 None admin] PATH_INFO: `/v3/projects/1e920620636443fb91e0138f970fd6e4` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:54.825130 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d641469-04a2-492f-ab7a-faf28fad9433 None admin] RBAC: Authorizing `identity:get_project(project_id=1e920620636443fb91e0138f970fd6e4)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:54.826565 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d641469-04a2-492f-ab7a-faf28fad9433 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:54.828184 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 581/1152] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:43:54 2022] GET /identity/v3/projects/1e920620636443fb91e0138f970fd6e4 => generated 311 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:43:54.834878 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d2b9dbb-0c5c-4c5e-b60d-3a06ccf85147 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:54.937206 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d2b9dbb-0c5c-4c5e-b60d-3a06ccf85147 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:54.973333 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea6700c-7e42-4203-9bc5-8e3f42c35490 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:54.974008 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea6700c-7e42-4203-9bc5-8e3f42c35490 None admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:54.974279 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea6700c-7e42-4203-9bc5-8e3f42c35490 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:54.974544 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea6700c-7e42-4203-9bc5-8e3f42c35490 None admin] PATH_INFO: `/v3/projects/1e920620636443fb91e0138f970fd6e4/users/81dce634acf24fcaa7295f08ec1b87dc/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:54.976553 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dea6700c-7e42-4203-9bc5-8e3f42c35490 None admin] RBAC: Authorizing `identity:create_grant(project_id=1e920620636443fb91e0138f970fd6e4, user_id=81dce634acf24fcaa7295f08ec1b87dc, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:54.978051 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dea6700c-7e42-4203-9bc5-8e3f42c35490 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:54.993732 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 572/1153] 10.222.0.52 () {58 vars in 1439 bytes} [Sun Sep 25 00:43:54 2022] PUT /identity/v3/projects/1e920620636443fb91e0138f970fd6e4/users/81dce634acf24fcaa7295f08ec1b87dc/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:43:57.173370 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d04c6e1-d3b3-4227-8fe1-dc8e28641d12 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:57.173958 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d04c6e1-d3b3-4227-8fe1-dc8e28641d12 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:57.173958 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d04c6e1-d3b3-4227-8fe1-dc8e28641d12 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:57.176072 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 582/1154] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:43:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:43:57.181238 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a64a5b-56b3-48d4-8a96-d505e2584537 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:57.181711 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a64a5b-56b3-48d4-8a96-d505e2584537 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:57.181840 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a64a5b-56b3-48d4-8a96-d505e2584537 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:57.200452 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-05a64a5b-56b3-48d4-8a96-d505e2584537 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:43:57.201666 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-05a64a5b-56b3-48d4-8a96-d505e2584537 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:43:57.221626 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:57.269027 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 573/1155] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:43:57 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:43:57.536981 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d04c6e1-d3b3-4227-8fe1-dc8e28641d12 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:57.548215 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:57.635515 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d04c6e1-d3b3-4227-8fe1-dc8e28641d12 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:57.667684 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8abd422a-079f-43d4-8719-47e1908fa919 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:57.668293 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8abd422a-079f-43d4-8719-47e1908fa919 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:57.668563 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8abd422a-079f-43d4-8719-47e1908fa919 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:57.668825 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8abd422a-079f-43d4-8719-47e1908fa919 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:57.679131 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8abd422a-079f-43d4-8719-47e1908fa919 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:43:57.683834 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-8abd422a-079f-43d4-8719-47e1908fa919 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:43:57.683834 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:43:57.683834 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:43:57.683834 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:43:57.683834 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:43:57.683834 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:43:57.683834 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:43:57.683834 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:43:57.683834 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:43:57.683834 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:43:57.683834 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:43:57.683834 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:43:57.683834 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:43:57.683834 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:43:57.683834 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:43:57.683834 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:43:57.684979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:43:57.684979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:43:57.684979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:43:57.684979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:43:57.684979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:43:57.684979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:43:57.684979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:43:57.684979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:43:57.684979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:43:57.684979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:43:57.684979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:43:57.684979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:43:57.684979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:43:57.684979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:43:57.684979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:43:57.684979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:43:57.686245 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:43:57.686245 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:43:57.686245 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:43:57.686245 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:43:57.686245 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:43:57.686245 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:43:57.686245 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:43:57.686245 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:43:57.686245 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:43:57.686245 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:43:57.686245 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:43:57.686245 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:43:57.686245 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 583/1156] 10.222.0.52 () {56 vars in 1096 bytes} [Sun Sep 25 00:43:57 2022] GET /identity/v3/roles/admin => generated 83 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:43:57.691881 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a64a5b-56b3-48d4-8a96-d505e2584537 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:57.790445 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a64a5b-56b3-48d4-8a96-d505e2584537 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:57.825318 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2a0e17-78c4-4332-923f-c46cd77d3c93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:57.825915 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2a0e17-78c4-4332-923f-c46cd77d3c93 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:57.826189 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2a0e17-78c4-4332-923f-c46cd77d3c93 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:57.826452 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2a0e17-78c4-4332-923f-c46cd77d3c93 None admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:57.827104 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b2a0e17-78c4-4332-923f-c46cd77d3c93 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:57.827398 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b2a0e17-78c4-4332-923f-c46cd77d3c93 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:57.828749 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b2a0e17-78c4-4332-923f-c46cd77d3c93 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:57.837162 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 574/1157] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:43:57 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:57.843186 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8abd422a-079f-43d4-8719-47e1908fa919 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:57.908194 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:57.941585 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8abd422a-079f-43d4-8719-47e1908fa919 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:57.973523 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699d00b5-4f4c-4ba2-b276-cd3b7f9d3d0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:57.974134 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699d00b5-4f4c-4ba2-b276-cd3b7f9d3d0d None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:57.974410 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699d00b5-4f4c-4ba2-b276-cd3b7f9d3d0d None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:57.974686 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699d00b5-4f4c-4ba2-b276-cd3b7f9d3d0d None admin] PATH_INFO: `/v3/users/81dce634acf24fcaa7295f08ec1b87dc` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:57.975793 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-699d00b5-4f4c-4ba2-b276-cd3b7f9d3d0d None admin] RBAC: Authorizing `identity:get_user(user_id=81dce634acf24fcaa7295f08ec1b87dc)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:57.977209 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-699d00b5-4f4c-4ba2-b276-cd3b7f9d3d0d None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:57.978802 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 584/1158] 10.222.0.52 () {56 vars in 1177 bytes} [Sun Sep 25 00:43:57 2022] GET /identity/v3/users/81dce634acf24fcaa7295f08ec1b87dc => generated 286 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:57.984613 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2a0e17-78c4-4332-923f-c46cd77d3c93 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:58.005221 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:58.084889 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2a0e17-78c4-4332-923f-c46cd77d3c93 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:58.117815 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168b8943-1106-4aca-85ba-58c00453f4f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:58.118419 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168b8943-1106-4aca-85ba-58c00453f4f9 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:58.118694 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168b8943-1106-4aca-85ba-58c00453f4f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:58.118956 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168b8943-1106-4aca-85ba-58c00453f4f9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:58.185133 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-168b8943-1106-4aca-85ba-58c00453f4f9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:58.186604 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-168b8943-1106-4aca-85ba-58c00453f4f9 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:58.204317 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:58.283793 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 575/1159] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:43:57 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 300 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 25 00:43:58.290363 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699d00b5-4f4c-4ba2-b276-cd3b7f9d3d0d None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:58.388795 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699d00b5-4f4c-4ba2-b276-cd3b7f9d3d0d None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:58.421867 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42b6eab6-2f4e-4d0b-9d0d-d0f86cc1ab1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:58.422475 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b6eab6-2f4e-4d0b-9d0d-d0f86cc1ab1e None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:58.422744 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b6eab6-2f4e-4d0b-9d0d-d0f86cc1ab1e None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:58.423006 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b6eab6-2f4e-4d0b-9d0d-d0f86cc1ab1e None admin] PATH_INFO: `/v3/projects/1e920620636443fb91e0138f970fd6e4` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:58.424096 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42b6eab6-2f4e-4d0b-9d0d-d0f86cc1ab1e None admin] RBAC: Authorizing `identity:get_project(project_id=1e920620636443fb91e0138f970fd6e4)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:58.426163 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42b6eab6-2f4e-4d0b-9d0d-d0f86cc1ab1e None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:58.427770 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 585/1160] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:43:58 2022] GET /identity/v3/projects/1e920620636443fb91e0138f970fd6e4 => generated 311 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:43:58.434029 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168b8943-1106-4aca-85ba-58c00453f4f9 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:43:58.532870 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168b8943-1106-4aca-85ba-58c00453f4f9 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:43:58.551033 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:43:58.568119 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2c701f7-b8f4-4637-815d-e85bafebae70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:43:58.568711 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c701f7-b8f4-4637-815d-e85bafebae70 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:43:58.568981 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c701f7-b8f4-4637-815d-e85bafebae70 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:43:58.569242 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c701f7-b8f4-4637-815d-e85bafebae70 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:43:58.569982 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2c701f7-b8f4-4637-815d-e85bafebae70 None admin] RBAC: Adding query filter params (role.id=2c242abf6ed8410ab0a4a354ca1794fb, scope.project.id=1e920620636443fb91e0138f970fd6e4, user.id=81dce634acf24fcaa7295f08ec1b87dc) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:43:58.570278 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2c701f7-b8f4-4637-815d-e85bafebae70 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:43:58.571689 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2c701f7-b8f4-4637-815d-e85bafebae70 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:43:58.577449 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 576/1161] 10.222.0.52 () {56 vars in 1374 bytes} [Sun Sep 25 00:43:58 2022] GET /identity/v3/role_assignments?user.id=81dce634acf24fcaa7295f08ec1b87dc&scope.project.id=1e920620636443fb91e0138f970fd6e4&role.id=2c242abf6ed8410ab0a4a354ca1794fb => generated 608 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:44:00.800361 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f908ab-34a8-49d9-873e-3f210cb0f695 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:44:00.800910 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f908ab-34a8-49d9-873e-3f210cb0f695 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:44:00.801047 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f908ab-34a8-49d9-873e-3f210cb0f695 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:44:00.802424 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 586/1162] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:44:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 00:44:00.811440 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23924b9-3555-40b0-931e-7ac600463a58 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:44:00.811978 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23924b9-3555-40b0-931e-7ac600463a58 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:44:00.812396 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23924b9-3555-40b0-931e-7ac600463a58 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:44:00.844695 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-f23924b9-3555-40b0-931e-7ac600463a58 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:44:00.848106 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-f23924b9-3555-40b0-931e-7ac600463a58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:44:00.970521 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 577/1163] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:44:00 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:44:01.237542 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f908ab-34a8-49d9-873e-3f210cb0f695 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:44:01.242618 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:44:01.347501 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f908ab-34a8-49d9-873e-3f210cb0f695 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:44:01.382718 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d71f05-94b4-479a-9235-2544ebf06282 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:44:01.383429 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d71f05-94b4-479a-9235-2544ebf06282 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:44:01.383785 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d71f05-94b4-479a-9235-2544ebf06282 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:44:01.384133 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d71f05-94b4-479a-9235-2544ebf06282 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:44:01.455350 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68d71f05-94b4-479a-9235-2544ebf06282 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:44:01.456971 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68d71f05-94b4-479a-9235-2544ebf06282 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:44:01.460223 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:44:01.566101 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 587/1164] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:44:01 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 329 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 25 00:44:01.572038 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f23924b9-3555-40b0-931e-7ac600463a58 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:44:01.683089 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f23924b9-3555-40b0-931e-7ac600463a58 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:44:01.698558 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:44:01.721492 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e078a9-e9f2-4fb1-ab0c-4b7ad850b7c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:44:01.722128 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e078a9-e9f2-4fb1-ab0c-4b7ad850b7c9 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:44:01.722396 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e078a9-e9f2-4fb1-ab0c-4b7ad850b7c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:44:01.722659 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e078a9-e9f2-4fb1-ab0c-4b7ad850b7c9 None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:44:01.729676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55e078a9-e9f2-4fb1-ab0c-4b7ad850b7c9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:44:01.731105 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55e078a9-e9f2-4fb1-ab0c-4b7ad850b7c9 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:44:01.743547 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-55e078a9-e9f2-4fb1-ab0c-4b7ad850b7c9 None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Sep 25 00:44:01.743547 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:44:01.743547 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Sep 25 00:44:01.743547 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Sep 25 00:44:01.743547 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 00:44:01.743547 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 00:44:01.743547 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 00:44:01.743547 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 00:44:01.743547 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Sep 25 00:44:01.743547 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:44:01.743547 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 00:44:01.743547 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:44:01.743547 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:44:01.743547 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:44:01.743547 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:44:01.743547 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:44:01.743547 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:44:01.745444 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:44:01.745444 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:44:01.745444 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:44:01.745444 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:44:01.745444 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:44:01.745444 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:44:01.745444 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:44:01.745444 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:44:01.745444 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:44:01.745444 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:44:01.745444 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:44:01.745444 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:44:01.745444 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:44:01.745444 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:44:01.745444 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:44:01.745444 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:44:01.747038 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:44:01.747038 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:44:01.747038 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:44:01.747038 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:44:01.747038 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:44:01.747038 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:44:01.747038 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:44:01.747038 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:44:01.747038 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:44:01.747038 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:44:01.747038 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Sep 25 00:44:01.747038 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:44:01.747038 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Sep 25 00:44:01.747038 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:44:01.747038 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 578/1165] 10.222.0.52 () {56 vars in 1117 bytes} [Sun Sep 25 00:44:01 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:44:01.750828 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d71f05-94b4-479a-9235-2544ebf06282 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:44:01.861780 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d71f05-94b4-479a-9235-2544ebf06282 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:44:01.865138 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:44:01.902637 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f63c642-5630-49b5-a774-dea13165753c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:44:01.903236 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f63c642-5630-49b5-a774-dea13165753c None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:44:01.903504 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f63c642-5630-49b5-a774-dea13165753c None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:44:01.903766 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f63c642-5630-49b5-a774-dea13165753c None admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:44:01.904390 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f63c642-5630-49b5-a774-dea13165753c None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:44:01.904683 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f63c642-5630-49b5-a774-dea13165753c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:44:01.906072 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f63c642-5630-49b5-a774-dea13165753c None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:44:01.913842 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 588/1166] 10.222.0.52 () {56 vars in 1115 bytes} [Sun Sep 25 00:44:01 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:44:04.927444 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6103cf-3299-4068-a727-6a7d6d40a446 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:44:04.927971 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6103cf-3299-4068-a727-6a7d6d40a446 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:44:04.927971 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6103cf-3299-4068-a727-6a7d6d40a446 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:44:04.929219 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 579/1167] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:44:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:44:04.934758 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d865a372-6be2-461a-8124-97f9840b71ed None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:44:04.935283 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d865a372-6be2-461a-8124-97f9840b71ed None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:44:04.935283 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d865a372-6be2-461a-8124-97f9840b71ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:44:04.957982 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-d865a372-6be2-461a-8124-97f9840b71ed None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:44:04.959168 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-d865a372-6be2-461a-8124-97f9840b71ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:44:05.028299 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 589/1168] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:44:04 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:44:05.346184 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b6103cf-3299-4068-a727-6a7d6d40a446 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:44:05.458835 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b6103cf-3299-4068-a727-6a7d6d40a446 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:44:05.495519 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d36ca49-205a-4670-b192-adf2ad904b0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:44:05.496130 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d36ca49-205a-4670-b192-adf2ad904b0b None admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:44:05.496400 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d36ca49-205a-4670-b192-adf2ad904b0b None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:44:05.496666 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d36ca49-205a-4670-b192-adf2ad904b0b None admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:44:05.497375 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d36ca49-205a-4670-b192-adf2ad904b0b None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:44:05.498766 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d36ca49-205a-4670-b192-adf2ad904b0b None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:44:05.516174 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 580/1169] 10.222.0.52 () {60 vars in 1138 bytes} [Sun Sep 25 00:44:05 2022] POST /identity/v3/domains => generated 267 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:44:07.818605 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0639e119-9964-4314-a771-4638b2004e3a None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:44:07.819127 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0639e119-9964-4314-a771-4638b2004e3a None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:44:07.819127 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0639e119-9964-4314-a771-4638b2004e3a None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:44:07.820369 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 590/1170] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:44:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 00:44:07.825975 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add34542-49a2-450a-8095-be54372d8efa None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:44:07.826460 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add34542-49a2-450a-8095-be54372d8efa None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:44:07.826589 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add34542-49a2-450a-8095-be54372d8efa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:44:07.845520 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-add34542-49a2-450a-8095-be54372d8efa None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:44:07.847070 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-add34542-49a2-450a-8095-be54372d8efa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:44:07.874297 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:44:07.921461 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 581/1171] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:44:07 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 25 00:44:08.203314 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0639e119-9964-4314-a771-4638b2004e3a None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:44:08.307122 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:44:08.312971 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0639e119-9964-4314-a771-4638b2004e3a None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:44:08.349237 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e661e0ba-bcc3-4349-861a-4b1e468c12cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:44:08.350065 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e661e0ba-bcc3-4349-861a-4b1e468c12cf None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:44:08.350378 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e661e0ba-bcc3-4349-861a-4b1e468c12cf None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:44:08.350684 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e661e0ba-bcc3-4349-861a-4b1e468c12cf None admin] PATH_INFO: `/v3/domains/6659f2d9be124ab48a2ae61665f10ce9` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:44:08.362478 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e661e0ba-bcc3-4349-861a-4b1e468c12cf None admin] RBAC: Authorizing `identity:get_domain(domain_id=6659f2d9be124ab48a2ae61665f10ce9)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:44:08.364018 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e661e0ba-bcc3-4349-861a-4b1e468c12cf None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:44:08.365827 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 591/1172] 10.222.0.52 () {56 vars in 1183 bytes} [Sun Sep 25 00:44:08 2022] GET /identity/v3/domains/6659f2d9be124ab48a2ae61665f10ce9 => generated 267 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:44:08.372904 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add34542-49a2-450a-8095-be54372d8efa None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:44:08.548271 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add34542-49a2-450a-8095-be54372d8efa None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:44:08.612908 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f657c5c-dd55-41c8-89ec-235c00d46c0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:44:08.613917 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f657c5c-dd55-41c8-89ec-235c00d46c0a None admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:44:08.614412 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f657c5c-dd55-41c8-89ec-235c00d46c0a None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:44:08.614885 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f657c5c-dd55-41c8-89ec-235c00d46c0a None admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:44:08.616053 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f657c5c-dd55-41c8-89ec-235c00d46c0a None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:44:08.618734 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f657c5c-dd55-41c8-89ec-235c00d46c0a None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:44:08.664471 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 582/1173] 10.222.0.52 () {60 vars in 1141 bytes} [Sun Sep 25 00:44:08 2022] POST /identity/v3/projects => generated 361 bytes in 293 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:44:11.091081 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af638214-9ea9-4da1-abec-71d72e4b5641 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:44:11.091678 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af638214-9ea9-4da1-abec-71d72e4b5641 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:44:11.091678 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af638214-9ea9-4da1-abec-71d72e4b5641 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:44:11.093241 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 592/1174] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:44:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 00:44:11.100051 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d54285c-3274-4615-9aec-6e59ae65cce9 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:44:11.100596 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d54285c-3274-4615-9aec-6e59ae65cce9 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:44:11.100596 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d54285c-3274-4615-9aec-6e59ae65cce9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:44:11.120472 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-8d54285c-3274-4615-9aec-6e59ae65cce9 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:44:11.121752 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-8d54285c-3274-4615-9aec-6e59ae65cce9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:44:11.133866 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:44:11.204219 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 583/1175] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:44:11 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 25 00:44:11.483445 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af638214-9ea9-4da1-abec-71d72e4b5641 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:44:11.597494 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af638214-9ea9-4da1-abec-71d72e4b5641 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:44:11.635210 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-839af728-adeb-4c4f-b07e-9e01c31b380a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:44:11.635842 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839af728-adeb-4c4f-b07e-9e01c31b380a None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:44:11.636116 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839af728-adeb-4c4f-b07e-9e01c31b380a None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:44:11.636384 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839af728-adeb-4c4f-b07e-9e01c31b380a None admin] PATH_INFO: `/v3/domains/6659f2d9be124ab48a2ae61665f10ce9` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:44:11.637568 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-839af728-adeb-4c4f-b07e-9e01c31b380a None admin] RBAC: Authorizing `identity:get_domain(domain_id=6659f2d9be124ab48a2ae61665f10ce9)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:44:11.639003 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-839af728-adeb-4c4f-b07e-9e01c31b380a None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:44:11.640610 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 593/1176] 10.222.0.52 () {56 vars in 1183 bytes} [Sun Sep 25 00:44:11 2022] GET /identity/v3/domains/6659f2d9be124ab48a2ae61665f10ce9 => generated 267 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:44:11.647694 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d54285c-3274-4615-9aec-6e59ae65cce9 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:44:11.766275 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d54285c-3274-4615-9aec-6e59ae65cce9 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:44:11.807673 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b52199de-bf86-4d67-a1db-6bc30648c652 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:44:11.808289 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52199de-bf86-4d67-a1db-6bc30648c652 None admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:44:11.808566 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52199de-bf86-4d67-a1db-6bc30648c652 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:44:11.808835 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52199de-bf86-4d67-a1db-6bc30648c652 None admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:44:11.809560 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b52199de-bf86-4d67-a1db-6bc30648c652 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:44:11.811067 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b52199de-bf86-4d67-a1db-6bc30648c652 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:44:11.832786 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:44:11.839778 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 584/1177] 10.222.0.52 () {60 vars in 1132 bytes} [Sun Sep 25 00:44:11 2022] POST /identity/v3/users => generated 311 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 00:44:14.182972 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e3da71-f0fa-4125-9cea-944f13530ea8 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:44:14.183537 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e3da71-f0fa-4125-9cea-944f13530ea8 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:44:14.183537 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e3da71-f0fa-4125-9cea-944f13530ea8 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:44:14.184848 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 594/1178] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:44:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 00:44:14.191024 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6836f85-6a21-4156-9891-78f0eadf6516 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:44:14.191528 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6836f85-6a21-4156-9891-78f0eadf6516 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:44:14.191677 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6836f85-6a21-4156-9891-78f0eadf6516 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:44:14.211691 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-d6836f85-6a21-4156-9891-78f0eadf6516 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:44:14.212898 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-d6836f85-6a21-4156-9891-78f0eadf6516 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:44:14.286069 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 585/1179] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:44:14 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 25 00:44:14.557034 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e3da71-f0fa-4125-9cea-944f13530ea8 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:44:14.586115 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:44:14.674698 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e3da71-f0fa-4125-9cea-944f13530ea8 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:44:14.712052 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe9d5db-6cfa-436b-a4ba-347d6bb9549d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:44:14.712682 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe9d5db-6cfa-436b-a4ba-347d6bb9549d None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:44:14.712954 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe9d5db-6cfa-436b-a4ba-347d6bb9549d None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:44:14.713217 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe9d5db-6cfa-436b-a4ba-347d6bb9549d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:44:14.725553 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe9d5db-6cfa-436b-a4ba-347d6bb9549d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:44:14.730645 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-efe9d5db-6cfa-436b-a4ba-347d6bb9549d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:44:14.730645 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:44:14.730645 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:44:14.730645 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:44:14.730645 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:44:14.730645 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:44:14.730645 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:44:14.730645 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:44:14.730645 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:44:14.730645 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:44:14.730645 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:44:14.730645 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:44:14.730645 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:44:14.730645 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:44:14.730645 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:44:14.730645 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:44:14.732523 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:44:14.732523 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:44:14.732523 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:44:14.732523 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:44:14.732523 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:44:14.732523 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:44:14.732523 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:44:14.732523 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:44:14.732523 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:44:14.732523 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:44:14.732523 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:44:14.732523 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:44:14.732523 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:44:14.732523 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:44:14.732523 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:44:14.732523 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:44:14.734244 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:44:14.734244 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:44:14.734244 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:44:14.734244 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:44:14.734244 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:44:14.734244 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:44:14.734244 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:44:14.734244 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:44:14.734244 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:44:14.734244 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:44:14.734244 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:44:14.734244 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:44:14.734244 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 595/1180] 10.222.0.52 () {56 vars in 1096 bytes} [Sun Sep 25 00:44:14 2022] GET /identity/v3/roles/admin => generated 83 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:44:14.740165 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6836f85-6a21-4156-9891-78f0eadf6516 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:44:14.849966 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6836f85-6a21-4156-9891-78f0eadf6516 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:44:14.881937 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:44:14.888953 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf5a8fa-f946-4345-afde-103736bb607b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:44:14.889662 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf5a8fa-f946-4345-afde-103736bb607b None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:44:14.889980 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf5a8fa-f946-4345-afde-103736bb607b None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:44:14.890289 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf5a8fa-f946-4345-afde-103736bb607b None admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:44:14.891009 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecf5a8fa-f946-4345-afde-103736bb607b None admin] RBAC: Adding query filter params (name=admin) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:44:14.891350 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecf5a8fa-f946-4345-afde-103736bb607b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:44:14.892840 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecf5a8fa-f946-4345-afde-103736bb607b None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:44:14.904486 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 586/1181] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:44:14 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:44:14.915477 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe9d5db-6cfa-436b-a4ba-347d6bb9549d None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:44:15.059593 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:44:15.076350 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe9d5db-6cfa-436b-a4ba-347d6bb9549d None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:44:15.112948 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baead482-5361-4886-a5ad-43e276b3f829 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:44:15.113686 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baead482-5361-4886-a5ad-43e276b3f829 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:44:15.114008 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baead482-5361-4886-a5ad-43e276b3f829 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:44:15.114321 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baead482-5361-4886-a5ad-43e276b3f829 None admin] PATH_INFO: `/v3/users/39069fe902674eb59766c008a2deb535` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:44:15.129437 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baead482-5361-4886-a5ad-43e276b3f829 None admin] RBAC: Authorizing `identity:get_user(user_id=39069fe902674eb59766c008a2deb535)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:44:15.130974 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baead482-5361-4886-a5ad-43e276b3f829 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:44:15.132759 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 596/1182] 10.222.0.52 () {56 vars in 1177 bytes} [Sun Sep 25 00:44:14 2022] GET /identity/v3/users/39069fe902674eb59766c008a2deb535 => generated 311 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:44:15.139321 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf5a8fa-f946-4345-afde-103736bb607b None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:44:15.253575 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf5a8fa-f946-4345-afde-103736bb607b None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:44:15.290375 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e45ed14-682e-4803-94d5-36ace36eced5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:44:15.290982 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e45ed14-682e-4803-94d5-36ace36eced5 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:44:15.291257 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e45ed14-682e-4803-94d5-36ace36eced5 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:44:15.291522 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e45ed14-682e-4803-94d5-36ace36eced5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:44:15.354566 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:44:15.366311 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e45ed14-682e-4803-94d5-36ace36eced5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:44:15.367918 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e45ed14-682e-4803-94d5-36ace36eced5 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:44:15.481189 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 587/1183] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:44:15 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 342 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 25 00:44:15.488269 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baead482-5361-4886-a5ad-43e276b3f829 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:44:15.602037 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baead482-5361-4886-a5ad-43e276b3f829 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:44:15.643206 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6552624e-c4f3-4c93-9faf-b43fbb5946f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:44:15.643873 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6552624e-c4f3-4c93-9faf-b43fbb5946f6 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:44:15.644192 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6552624e-c4f3-4c93-9faf-b43fbb5946f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:44:15.644502 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6552624e-c4f3-4c93-9faf-b43fbb5946f6 None admin] PATH_INFO: `/v3/projects/cc4484a3f1c844bbbdf66977840f0fbc` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:44:15.645827 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6552624e-c4f3-4c93-9faf-b43fbb5946f6 None admin] RBAC: Authorizing `identity:get_project(project_id=cc4484a3f1c844bbbdf66977840f0fbc)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:44:15.647311 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6552624e-c4f3-4c93-9faf-b43fbb5946f6 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:44:15.649055 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 597/1184] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:44:15 2022] GET /identity/v3/projects/cc4484a3f1c844bbbdf66977840f0fbc => generated 361 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:44:15.659779 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e45ed14-682e-4803-94d5-36ace36eced5 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:44:15.787981 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:44:15.802308 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e45ed14-682e-4803-94d5-36ace36eced5 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:44:15.846890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d47ecd-8fa3-49c9-8a16-e3366ba74999 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:44:15.847511 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d47ecd-8fa3-49c9-8a16-e3366ba74999 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:44:15.847787 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d47ecd-8fa3-49c9-8a16-e3366ba74999 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:44:15.848052 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d47ecd-8fa3-49c9-8a16-e3366ba74999 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:44:15.848788 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d47ecd-8fa3-49c9-8a16-e3366ba74999 None admin] RBAC: Adding query filter params (role.id=2c242abf6ed8410ab0a4a354ca1794fb, scope.project.id=cc4484a3f1c844bbbdf66977840f0fbc, user.id=39069fe902674eb59766c008a2deb535) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:44:15.849097 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d47ecd-8fa3-49c9-8a16-e3366ba74999 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:44:15.850611 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d47ecd-8fa3-49c9-8a16-e3366ba74999 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:44:15.858585 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 588/1185] 10.222.0.52 () {56 vars in 1374 bytes} [Sun Sep 25 00:44:15 2022] GET /identity/v3/role_assignments?user.id=39069fe902674eb59766c008a2deb535&scope.project.id=cc4484a3f1c844bbbdf66977840f0fbc&role.id=2c242abf6ed8410ab0a4a354ca1794fb => generated 259 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:44:18.644996 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240b65be-d757-4e7e-85ca-f0f0d8bafd72 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:44:18.645926 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240b65be-d757-4e7e-85ca-f0f0d8bafd72 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:44:18.645926 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240b65be-d757-4e7e-85ca-f0f0d8bafd72 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:44:18.646870 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 598/1186] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:44:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 00:44:18.652799 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735cbc93-d51a-471c-a6c0-825497927c01 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:44:18.653429 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735cbc93-d51a-471c-a6c0-825497927c01 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:44:18.653429 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735cbc93-d51a-471c-a6c0-825497927c01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:44:18.673314 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-735cbc93-d51a-471c-a6c0-825497927c01 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:44:18.674817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-735cbc93-d51a-471c-a6c0-825497927c01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:44:18.757580 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 589/1187] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:44:18 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 25 00:44:19.039421 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-240b65be-d757-4e7e-85ca-f0f0d8bafd72 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:44:19.056310 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:44:19.157642 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-240b65be-d757-4e7e-85ca-f0f0d8bafd72 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:44:19.197815 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961565cf-e47f-44ba-9bf6-63f109bb8f39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:44:19.198497 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961565cf-e47f-44ba-9bf6-63f109bb8f39 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:44:19.198813 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961565cf-e47f-44ba-9bf6-63f109bb8f39 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:44:19.199122 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961565cf-e47f-44ba-9bf6-63f109bb8f39 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:44:19.210832 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-961565cf-e47f-44ba-9bf6-63f109bb8f39 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:44:19.215759 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-961565cf-e47f-44ba-9bf6-63f109bb8f39 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:44:19.215759 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:44:19.215759 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:44:19.215759 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:44:19.215759 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:44:19.215759 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:44:19.215759 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:44:19.215759 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:44:19.215759 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:44:19.215759 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:44:19.215759 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:44:19.215759 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:44:19.215759 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:44:19.215759 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:44:19.215759 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:44:19.215759 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:44:19.216981 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:44:19.216981 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:44:19.216981 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:44:19.216981 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:44:19.216981 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:44:19.216981 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:44:19.216981 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:44:19.216981 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:44:19.216981 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:44:19.216981 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:44:19.216981 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:44:19.216981 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:44:19.216981 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:44:19.216981 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:44:19.216981 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:44:19.216981 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:44:19.218465 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:44:19.218465 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:44:19.218465 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:44:19.218465 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:44:19.218465 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:44:19.218465 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:44:19.218465 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:44:19.218465 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:44:19.218465 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:44:19.218465 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:44:19.218465 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:44:19.218465 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:44:19.218465 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 599/1188] 10.222.0.52 () {56 vars in 1096 bytes} [Sun Sep 25 00:44:19 2022] GET /identity/v3/roles/admin => generated 83 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:44:19.225631 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735cbc93-d51a-471c-a6c0-825497927c01 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:44:19.344554 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:44:19.363126 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735cbc93-d51a-471c-a6c0-825497927c01 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:44:19.403925 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-280de252-0ad3-4504-a480-23ec3fa5ca75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:44:19.404589 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280de252-0ad3-4504-a480-23ec3fa5ca75 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:44:19.404905 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280de252-0ad3-4504-a480-23ec3fa5ca75 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:44:19.405214 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280de252-0ad3-4504-a480-23ec3fa5ca75 None admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:44:19.405960 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-280de252-0ad3-4504-a480-23ec3fa5ca75 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:44:19.406294 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-280de252-0ad3-4504-a480-23ec3fa5ca75 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:44:19.407794 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-280de252-0ad3-4504-a480-23ec3fa5ca75 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:44:19.419299 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 590/1189] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:44:19 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:44:19.429287 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961565cf-e47f-44ba-9bf6-63f109bb8f39 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:44:19.513152 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:44:19.551256 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961565cf-e47f-44ba-9bf6-63f109bb8f39 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:44:19.589194 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e811f2-d1b5-4aa0-a5e2-95c19c9b4e7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:44:19.589847 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e811f2-d1b5-4aa0-a5e2-95c19c9b4e7c None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:44:19.590116 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e811f2-d1b5-4aa0-a5e2-95c19c9b4e7c None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:44:19.590377 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e811f2-d1b5-4aa0-a5e2-95c19c9b4e7c None admin] PATH_INFO: `/v3/users/39069fe902674eb59766c008a2deb535` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:44:19.591516 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e811f2-d1b5-4aa0-a5e2-95c19c9b4e7c None admin] RBAC: Authorizing `identity:get_user(user_id=39069fe902674eb59766c008a2deb535)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:44:19.592948 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e811f2-d1b5-4aa0-a5e2-95c19c9b4e7c None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:44:19.594690 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 600/1190] 10.222.0.52 () {56 vars in 1177 bytes} [Sun Sep 25 00:44:19 2022] GET /identity/v3/users/39069fe902674eb59766c008a2deb535 => generated 311 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:44:19.603759 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-280de252-0ad3-4504-a480-23ec3fa5ca75 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:44:19.713872 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-280de252-0ad3-4504-a480-23ec3fa5ca75 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:44:19.752703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c4c20a9-f5fc-42fc-b536-3527e7616bbc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:44:19.753438 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4c20a9-f5fc-42fc-b536-3527e7616bbc None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:44:19.753716 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4c20a9-f5fc-42fc-b536-3527e7616bbc None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:44:19.753983 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4c20a9-f5fc-42fc-b536-3527e7616bbc None admin] PATH_INFO: `/v3/projects/cc4484a3f1c844bbbdf66977840f0fbc` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:44:19.755128 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c4c20a9-f5fc-42fc-b536-3527e7616bbc None admin] RBAC: Authorizing `identity:get_project(project_id=cc4484a3f1c844bbbdf66977840f0fbc)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:44:19.756559 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c4c20a9-f5fc-42fc-b536-3527e7616bbc None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:44:19.758253 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 591/1191] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:44:19 2022] GET /identity/v3/projects/cc4484a3f1c844bbbdf66977840f0fbc => generated 361 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:44:19.764557 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e811f2-d1b5-4aa0-a5e2-95c19c9b4e7c None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:44:19.873402 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e811f2-d1b5-4aa0-a5e2-95c19c9b4e7c None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:44:19.910354 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1ac6d9-73d3-4bbf-963a-fd131d4d2696 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:44:19.911013 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1ac6d9-73d3-4bbf-963a-fd131d4d2696 None admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:44:19.911281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1ac6d9-73d3-4bbf-963a-fd131d4d2696 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:44:19.911547 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1ac6d9-73d3-4bbf-963a-fd131d4d2696 None admin] PATH_INFO: `/v3/projects/cc4484a3f1c844bbbdf66977840f0fbc/users/39069fe902674eb59766c008a2deb535/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:44:19.913550 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e1ac6d9-73d3-4bbf-963a-fd131d4d2696 None admin] RBAC: Authorizing `identity:create_grant(project_id=cc4484a3f1c844bbbdf66977840f0fbc, user_id=39069fe902674eb59766c008a2deb535, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:44:19.914977 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e1ac6d9-73d3-4bbf-963a-fd131d4d2696 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:44:19.925985 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:44:19.931134 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 601/1192] 10.222.0.52 () {58 vars in 1439 bytes} [Sun Sep 25 00:44:19 2022] PUT /identity/v3/projects/cc4484a3f1c844bbbdf66977840f0fbc/users/39069fe902674eb59766c008a2deb535/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:44:22.209491 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1dc90e2-3b46-4f39-a524-273645ee53b7 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:44:22.210057 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1dc90e2-3b46-4f39-a524-273645ee53b7 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:44:22.210181 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1dc90e2-3b46-4f39-a524-273645ee53b7 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:44:22.211585 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 592/1193] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:44:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 00:44:22.217636 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51a00ff-3e52-44df-a1a3-ae635fd54ca8 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:44:22.218216 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51a00ff-3e52-44df-a1a3-ae635fd54ca8 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:44:22.218216 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51a00ff-3e52-44df-a1a3-ae635fd54ca8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:44:22.238619 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-a51a00ff-3e52-44df-a1a3-ae635fd54ca8 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:44:22.239837 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-a51a00ff-3e52-44df-a1a3-ae635fd54ca8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:44:22.321020 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 602/1194] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:44:22 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 25 00:44:22.599994 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1dc90e2-3b46-4f39-a524-273645ee53b7 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:44:22.620545 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:44:22.712662 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1dc90e2-3b46-4f39-a524-273645ee53b7 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:44:22.748703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477a7896-9256-4e72-9031-472b56f0d2a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:44:22.749236 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477a7896-9256-4e72-9031-472b56f0d2a3 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:44:22.749543 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477a7896-9256-4e72-9031-472b56f0d2a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:44:22.749811 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477a7896-9256-4e72-9031-472b56f0d2a3 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:44:22.761645 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-477a7896-9256-4e72-9031-472b56f0d2a3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:44:22.766493 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-477a7896-9256-4e72-9031-472b56f0d2a3 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:44:22.766493 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:44:22.766493 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:44:22.766493 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:44:22.766493 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:44:22.766493 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:44:22.766493 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:44:22.766493 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:44:22.766493 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:44:22.766493 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:44:22.766493 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:44:22.766493 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:44:22.766493 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:44:22.766493 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:44:22.766493 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:44:22.766493 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:44:22.768865 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:44:22.768865 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:44:22.768865 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:44:22.768865 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:44:22.768865 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:44:22.768865 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:44:22.768865 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:44:22.768865 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:44:22.768865 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:44:22.768865 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:44:22.768865 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:44:22.768865 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:44:22.768865 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:44:22.768865 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:44:22.768865 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:44:22.768865 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:44:22.770585 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:44:22.770585 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:44:22.770585 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:44:22.770585 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:44:22.770585 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:44:22.770585 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:44:22.770585 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:44:22.770585 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:44:22.770585 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:44:22.770585 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:44:22.770585 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:44:22.770585 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:44:22.770585 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 593/1195] 10.222.0.52 () {56 vars in 1096 bytes} [Sun Sep 25 00:44:22 2022] GET /identity/v3/roles/admin => generated 83 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:44:22.774583 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a51a00ff-3e52-44df-a1a3-ae635fd54ca8 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:44:22.885450 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a51a00ff-3e52-44df-a1a3-ae635fd54ca8 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:44:22.917438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:44:22.924601 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d6c3a4-4281-489c-a794-904bb24a8681 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:44:22.925220 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d6c3a4-4281-489c-a794-904bb24a8681 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:44:22.925528 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d6c3a4-4281-489c-a794-904bb24a8681 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:44:22.925797 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d6c3a4-4281-489c-a794-904bb24a8681 None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:44:22.926492 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85d6c3a4-4281-489c-a794-904bb24a8681 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:44:22.926803 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85d6c3a4-4281-489c-a794-904bb24a8681 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:44:22.928212 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85d6c3a4-4281-489c-a794-904bb24a8681 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:44:22.937503 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 603/1196] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:44:22 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:44:22.944351 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477a7896-9256-4e72-9031-472b56f0d2a3 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:44:23.038196 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:44:23.077356 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477a7896-9256-4e72-9031-472b56f0d2a3 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:44:23.138423 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f829f7a-2615-4af4-a1b2-db5a07f336ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:44:23.139458 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f829f7a-2615-4af4-a1b2-db5a07f336ed None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:44:23.139931 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f829f7a-2615-4af4-a1b2-db5a07f336ed None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:44:23.140400 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f829f7a-2615-4af4-a1b2-db5a07f336ed None admin] PATH_INFO: `/v3/users/39069fe902674eb59766c008a2deb535` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:44:23.142277 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f829f7a-2615-4af4-a1b2-db5a07f336ed None admin] RBAC: Authorizing `identity:get_user(user_id=39069fe902674eb59766c008a2deb535)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:44:23.143753 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f829f7a-2615-4af4-a1b2-db5a07f336ed None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:44:23.145495 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 594/1197] 10.222.0.52 () {56 vars in 1177 bytes} [Sun Sep 25 00:44:22 2022] GET /identity/v3/users/39069fe902674eb59766c008a2deb535 => generated 311 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:44:23.151550 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d6c3a4-4281-489c-a794-904bb24a8681 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:44:23.267933 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d6c3a4-4281-489c-a794-904bb24a8681 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:44:23.304156 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd4042aa-7f08-4a04-8eaa-b2ac3e8b5f8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:44:23.304675 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd4042aa-7f08-4a04-8eaa-b2ac3e8b5f8c None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:44:23.304951 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd4042aa-7f08-4a04-8eaa-b2ac3e8b5f8c None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:44:23.305219 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd4042aa-7f08-4a04-8eaa-b2ac3e8b5f8c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:44:23.367691 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:44:23.378544 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd4042aa-7f08-4a04-8eaa-b2ac3e8b5f8c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:44:23.380052 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd4042aa-7f08-4a04-8eaa-b2ac3e8b5f8c None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:44:23.488109 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 604/1198] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:44:23 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 337 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 25 00:44:23.494286 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f829f7a-2615-4af4-a1b2-db5a07f336ed None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:44:23.604244 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f829f7a-2615-4af4-a1b2-db5a07f336ed None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:44:23.655241 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba7f79cc-8230-45c2-bdb1-4a6b7573bc8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:44:23.656279 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7f79cc-8230-45c2-bdb1-4a6b7573bc8a None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:44:23.656766 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7f79cc-8230-45c2-bdb1-4a6b7573bc8a None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:44:23.657246 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7f79cc-8230-45c2-bdb1-4a6b7573bc8a None admin] PATH_INFO: `/v3/projects/cc4484a3f1c844bbbdf66977840f0fbc` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:44:23.659807 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba7f79cc-8230-45c2-bdb1-4a6b7573bc8a None admin] RBAC: Authorizing `identity:get_project(project_id=cc4484a3f1c844bbbdf66977840f0fbc)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:44:23.662494 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba7f79cc-8230-45c2-bdb1-4a6b7573bc8a None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:44:23.665518 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 595/1199] 10.222.0.52 () {56 vars in 1186 bytes} [Sun Sep 25 00:44:23 2022] GET /identity/v3/projects/cc4484a3f1c844bbbdf66977840f0fbc => generated 361 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:44:23.671916 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd4042aa-7f08-4a04-8eaa-b2ac3e8b5f8c None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:44:23.768044 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:44:23.782336 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd4042aa-7f08-4a04-8eaa-b2ac3e8b5f8c None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:44:23.829308 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9693348c-8a34-461a-828d-87701bf3ef29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:44:23.830098 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9693348c-8a34-461a-828d-87701bf3ef29 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:44:23.830370 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9693348c-8a34-461a-828d-87701bf3ef29 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:44:23.830633 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9693348c-8a34-461a-828d-87701bf3ef29 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:44:23.831345 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9693348c-8a34-461a-828d-87701bf3ef29 None admin] RBAC: Adding query filter params (role.id=2c242abf6ed8410ab0a4a354ca1794fb, scope.project.id=cc4484a3f1c844bbbdf66977840f0fbc, user.id=39069fe902674eb59766c008a2deb535) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:44:23.831644 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9693348c-8a34-461a-828d-87701bf3ef29 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:44:23.833084 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9693348c-8a34-461a-828d-87701bf3ef29 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:44:23.839127 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 605/1200] 10.222.0.52 () {56 vars in 1374 bytes} [Sun Sep 25 00:44:23 2022] GET /identity/v3/role_assignments?user.id=39069fe902674eb59766c008a2deb535&scope.project.id=cc4484a3f1c844bbbdf66977840f0fbc&role.id=2c242abf6ed8410ab0a4a354ca1794fb => generated 608 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:44:33.648530 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=94 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:44:33.836675 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=22 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:24.045810 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7865191d-9c0c-4fc1-bd13-aabdda0ea4f3 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:24.045810 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7865191d-9c0c-4fc1-bd13-aabdda0ea4f3 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:24.047601 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7865191d-9c0c-4fc1-bd13-aabdda0ea4f3 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:24.050998 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 596/1201] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:45:24 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:45:24.059001 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ada603-1b35-488e-a56e-9488b33ca2f7 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:24.059828 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ada603-1b35-488e-a56e-9488b33ca2f7 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:24.059993 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ada603-1b35-488e-a56e-9488b33ca2f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:24.090054 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-77ada603-1b35-488e-a56e-9488b33ca2f7 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:45:24.092942 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-77ada603-1b35-488e-a56e-9488b33ca2f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:45:24.222350 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 606/1202] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:45:24 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 25 00:45:24.561890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7865191d-9c0c-4fc1-bd13-aabdda0ea4f3 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:24.760929 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7865191d-9c0c-4fc1-bd13-aabdda0ea4f3 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:24.820663 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-190c0134-61b4-4d34-8f07-ba00bbc35c23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:24.821728 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190c0134-61b4-4d34-8f07-ba00bbc35c23 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:24.822396 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190c0134-61b4-4d34-8f07-ba00bbc35c23 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:24.822830 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190c0134-61b4-4d34-8f07-ba00bbc35c23 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:24.841177 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-190c0134-61b4-4d34-8f07-ba00bbc35c23 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:24.844122 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-190c0134-61b4-4d34-8f07-ba00bbc35c23 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:24.879147 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-190c0134-61b4-4d34-8f07-ba00bbc35c23 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:45:24.879147 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:45:24.879147 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:45:24.879147 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:45:24.879147 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:45:24.879147 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:45:24.879147 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:45:24.879147 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:45:24.879147 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:45:24.879147 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:45:24.879147 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:45:24.879147 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:45:24.879147 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:45:24.879147 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:45:24.879147 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:45:24.879147 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:45:24.880933 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:45:24.880933 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:45:24.880933 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:45:24.880933 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:45:24.880933 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:45:24.880933 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:45:24.880933 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:45:24.880933 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:45:24.880933 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:45:24.880933 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:45:24.880933 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:45:24.880933 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:45:24.880933 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:45:24.880933 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:45:24.880933 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:45:24.880933 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:45:24.880933 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:45:24.882996 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:45:24.882996 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:45:24.882996 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:45:24.882996 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 597/1203] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:45:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 320 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:45:24.891308 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ada603-1b35-488e-a56e-9488b33ca2f7 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:25.085289 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ada603-1b35-488e-a56e-9488b33ca2f7 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:25.144433 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:25.153133 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a4186f-0de6-4d49-9eda-854252fb7f47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:25.154401 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a4186f-0de6-4d49-9eda-854252fb7f47 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:25.155536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a4186f-0de6-4d49-9eda-854252fb7f47 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:25.156310 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a4186f-0de6-4d49-9eda-854252fb7f47 None admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:25.157754 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02a4186f-0de6-4d49-9eda-854252fb7f47 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:45:25.158339 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02a4186f-0de6-4d49-9eda-854252fb7f47 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:25.161147 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02a4186f-0de6-4d49-9eda-854252fb7f47 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:25.188059 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 607/1204] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:45:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 297 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:45:25.197252 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-190c0134-61b4-4d34-8f07-ba00bbc35c23 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:25.309322 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:25.422888 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-190c0134-61b4-4d34-8f07-ba00bbc35c23 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:25.472857 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e20f450c-a262-41cf-b77f-c75f87687f0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:25.473623 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20f450c-a262-41cf-b77f-c75f87687f0e None admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:25.473905 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20f450c-a262-41cf-b77f-c75f87687f0e None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:25.474161 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20f450c-a262-41cf-b77f-c75f87687f0e None admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:25.474918 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e20f450c-a262-41cf-b77f-c75f87687f0e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:25.476704 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e20f450c-a262-41cf-b77f-c75f87687f0e None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:25.514673 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 598/1205] 10.222.0.52 () {60 vars in 1132 bytes} [Sun Sep 25 00:45:25 2022] POST /identity/v3/users => generated 251 bytes in 317 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:45:28.401133 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c98a23a-3533-47f0-8fd8-8ddeeabde04c None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:28.401133 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c98a23a-3533-47f0-8fd8-8ddeeabde04c None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:28.401133 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c98a23a-3533-47f0-8fd8-8ddeeabde04c None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:28.401133 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 608/1206] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:45:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 00:45:28.410708 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cbb2aaa-74de-4d3b-9adc-a62826413eaf None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:28.411273 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cbb2aaa-74de-4d3b-9adc-a62826413eaf None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:28.411361 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cbb2aaa-74de-4d3b-9adc-a62826413eaf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:28.435085 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-5cbb2aaa-74de-4d3b-9adc-a62826413eaf None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:45:28.436661 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-5cbb2aaa-74de-4d3b-9adc-a62826413eaf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:45:28.533663 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 599/1207] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:45:28 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:45:28.822432 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c98a23a-3533-47f0-8fd8-8ddeeabde04c None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:29.018224 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c98a23a-3533-47f0-8fd8-8ddeeabde04c None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:29.093366 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06cb22c4-fdfb-4865-89e2-f6bc79bfad0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:29.093903 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06cb22c4-fdfb-4865-89e2-f6bc79bfad0c None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:29.094188 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06cb22c4-fdfb-4865-89e2-f6bc79bfad0c None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:29.094439 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06cb22c4-fdfb-4865-89e2-f6bc79bfad0c None admin] PATH_INFO: `/v3/roles/service` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:29.110248 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06cb22c4-fdfb-4865-89e2-f6bc79bfad0c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:45:29.116367 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-06cb22c4-fdfb-4865-89e2-f6bc79bfad0c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 25 00:45:29.116367 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:45:29.116367 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:45:29.116367 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:45:29.116367 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:45:29.116367 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:45:29.116367 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:45:29.116367 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:45:29.116367 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:45:29.116367 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:45:29.116367 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:45:29.116367 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:45:29.116367 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:45:29.116367 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:45:29.116367 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:45:29.116367 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:45:29.121801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:45:29.121801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:45:29.121801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:45:29.121801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:45:29.121801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:45:29.121801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:45:29.121801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:45:29.121801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:45:29.121801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:45:29.121801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:45:29.121801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:45:29.121801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:45:29.121801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:45:29.121801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:45:29.121801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:45:29.121801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:45:29.123796 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:45:29.123796 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:45:29.123796 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:45:29.123796 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:45:29.123796 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:45:29.123796 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:45:29.123796 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:45:29.123796 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:45:29.123796 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:45:29.123796 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:45:29.123796 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 25 00:45:29.123796 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:45:29.123796 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 609/1208] 10.222.0.52 () {56 vars in 1102 bytes} [Sun Sep 25 00:45:28 2022] GET /identity/v3/roles/service => generated 85 bytes in 296 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:45:29.129688 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cbb2aaa-74de-4d3b-9adc-a62826413eaf None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:29.181454 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:29.269763 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cbb2aaa-74de-4d3b-9adc-a62826413eaf None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:29.316585 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caa6cc09-2161-44af-adcf-12b94b41fc2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:29.317198 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa6cc09-2161-44af-adcf-12b94b41fc2d None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:29.317508 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa6cc09-2161-44af-adcf-12b94b41fc2d None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:29.317776 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa6cc09-2161-44af-adcf-12b94b41fc2d None admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:29.318549 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caa6cc09-2161-44af-adcf-12b94b41fc2d None admin] RBAC: Adding query filter params (name=service) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:45:29.318971 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caa6cc09-2161-44af-adcf-12b94b41fc2d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:29.320616 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caa6cc09-2161-44af-adcf-12b94b41fc2d None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:29.331362 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 600/1209] 10.222.0.52 () {56 vars in 1103 bytes} [Sun Sep 25 00:45:29 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:45:29.338280 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06cb22c4-fdfb-4865-89e2-f6bc79bfad0c None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:29.412523 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:29.487038 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06cb22c4-fdfb-4865-89e2-f6bc79bfad0c None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:29.565551 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1067d5bd-ea96-4222-bc64-f46ca91abfa0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:29.566747 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1067d5bd-ea96-4222-bc64-f46ca91abfa0 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:29.567295 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1067d5bd-ea96-4222-bc64-f46ca91abfa0 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:29.567876 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1067d5bd-ea96-4222-bc64-f46ca91abfa0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:29.588786 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1067d5bd-ea96-4222-bc64-f46ca91abfa0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:29.590690 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1067d5bd-ea96-4222-bc64-f46ca91abfa0 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:29.620953 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-1067d5bd-ea96-4222-bc64-f46ca91abfa0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:45:29.620953 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:45:29.620953 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:45:29.620953 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:45:29.620953 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:45:29.620953 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:45:29.620953 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:45:29.620953 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:45:29.620953 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:45:29.620953 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:45:29.620953 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:45:29.620953 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:45:29.620953 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:45:29.620953 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:45:29.620953 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:45:29.620953 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:45:29.622848 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:45:29.622848 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:45:29.622848 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:45:29.622848 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:45:29.622848 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:45:29.622848 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:45:29.622848 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:45:29.622848 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:45:29.622848 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:45:29.622848 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:45:29.622848 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:45:29.622848 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:45:29.622848 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:45:29.622848 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:45:29.622848 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:45:29.622848 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:45:29.622848 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:45:29.624807 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:45:29.624807 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:45:29.624807 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:45:29.624807 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 610/1210] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:45:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 286 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:45:29.634370 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caa6cc09-2161-44af-adcf-12b94b41fc2d None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:29.786770 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:29.790757 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caa6cc09-2161-44af-adcf-12b94b41fc2d None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:29.852205 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc19ac68-eb8c-48da-9bbc-125b96f863ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:29.853453 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc19ac68-eb8c-48da-9bbc-125b96f863ae None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:29.853793 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc19ac68-eb8c-48da-9bbc-125b96f863ae None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:29.854050 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc19ac68-eb8c-48da-9bbc-125b96f863ae None admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:29.855072 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc19ac68-eb8c-48da-9bbc-125b96f863ae None admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:45:29.855476 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc19ac68-eb8c-48da-9bbc-125b96f863ae None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:29.857356 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc19ac68-eb8c-48da-9bbc-125b96f863ae None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:29.876714 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 601/1211] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:45:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:45:29.887374 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1067d5bd-ea96-4222-bc64-f46ca91abfa0 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:30.065141 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1067d5bd-ea96-4222-bc64-f46ca91abfa0 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:30.068129 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:30.132194 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53af513a-38ba-4e45-b69d-9419a6946c8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:30.132980 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53af513a-38ba-4e45-b69d-9419a6946c8d None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:30.133233 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53af513a-38ba-4e45-b69d-9419a6946c8d None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:30.133729 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53af513a-38ba-4e45-b69d-9419a6946c8d None admin] PATH_INFO: `/v3/users/placement` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:30.145163 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53af513a-38ba-4e45-b69d-9419a6946c8d None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:30.146708 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53af513a-38ba-4e45-b69d-9419a6946c8d None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:30.161405 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-53af513a-38ba-4e45-b69d-9419a6946c8d None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 25 00:45:30.161405 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:45:30.161405 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:45:30.161405 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:45:30.161405 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:45:30.161405 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:45:30.161405 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:45:30.161405 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:45:30.161405 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:45:30.161405 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:45:30.161405 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:45:30.161405 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:45:30.161405 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 00:45:30.161405 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 00:45:30.161405 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 00:45:30.161405 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 00:45:30.163031 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:45:30.163031 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:45:30.163031 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 00:45:30.163031 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:45:30.163031 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 00:45:30.163031 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:45:30.163031 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:45:30.163031 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:45:30.163031 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:45:30.163031 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:45:30.163031 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:45:30.163031 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:45:30.163031 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:45:30.163031 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:45:30.163031 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:45:30.163031 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:45:30.163031 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:45:30.163031 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:45:30.164732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:45:30.164732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:45:30.164732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 00:45:30.164732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 00:45:30.164732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 00:45:30.164732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 00:45:30.164732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 00:45:30.164732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 00:45:30.164732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 25 00:45:30.164732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:45:30.164732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 611/1212] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:45:29 2022] GET /identity/v3/users/placement => generated 87 bytes in 276 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:45:30.169667 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc19ac68-eb8c-48da-9bbc-125b96f863ae None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:30.367075 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc19ac68-eb8c-48da-9bbc-125b96f863ae None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:30.431579 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b31f74a-5f5a-4cbf-9dc6-a4f122e0ebeb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:30.432259 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b31f74a-5f5a-4cbf-9dc6-a4f122e0ebeb None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:30.432573 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b31f74a-5f5a-4cbf-9dc6-a4f122e0ebeb None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:30.433034 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b31f74a-5f5a-4cbf-9dc6-a4f122e0ebeb None admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:30.434076 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b31f74a-5f5a-4cbf-9dc6-a4f122e0ebeb None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:45:30.434512 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b31f74a-5f5a-4cbf-9dc6-a4f122e0ebeb None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:30.437961 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b31f74a-5f5a-4cbf-9dc6-a4f122e0ebeb None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:30.464578 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 602/1213] 10.222.0.52 () {56 vars in 1143 bytes} [Sun Sep 25 00:45:30 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 296 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:45:30.471395 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53af513a-38ba-4e45-b69d-9419a6946c8d None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:30.601630 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53af513a-38ba-4e45-b69d-9419a6946c8d None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:30.644646 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed1fee4-1576-4f30-8ec3-c0fd02b9ffcf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:30.645166 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed1fee4-1576-4f30-8ec3-c0fd02b9ffcf None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:30.645471 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed1fee4-1576-4f30-8ec3-c0fd02b9ffcf None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:30.645742 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed1fee4-1576-4f30-8ec3-c0fd02b9ffcf None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:30.680692 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:30.724689 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ed1fee4-1576-4f30-8ec3-c0fd02b9ffcf None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:30.726179 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ed1fee4-1576-4f30-8ec3-c0fd02b9ffcf None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:30.843167 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 612/1214] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:45:30 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 25 00:45:30.849657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b31f74a-5f5a-4cbf-9dc6-a4f122e0ebeb None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:30.878643 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:31.010371 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b31f74a-5f5a-4cbf-9dc6-a4f122e0ebeb None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:31.066269 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cd82f69-563b-492b-b31c-8b1fa48e8e10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:31.067265 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd82f69-563b-492b-b31c-8b1fa48e8e10 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:31.067749 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd82f69-563b-492b-b31c-8b1fa48e8e10 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:31.068256 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd82f69-563b-492b-b31c-8b1fa48e8e10 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:31.090259 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cd82f69-563b-492b-b31c-8b1fa48e8e10 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:31.092741 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cd82f69-563b-492b-b31c-8b1fa48e8e10 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:31.116145 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-2cd82f69-563b-492b-b31c-8b1fa48e8e10 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:45:31.116145 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:45:31.116145 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:45:31.116145 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:45:31.116145 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:45:31.116145 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:45:31.116145 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:45:31.116145 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:45:31.116145 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:45:31.116145 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:45:31.116145 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:45:31.116145 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:45:31.116145 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:45:31.116145 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:45:31.116145 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:45:31.116145 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:45:31.117919 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:45:31.117919 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:45:31.117919 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:45:31.117919 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:45:31.117919 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:45:31.117919 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:45:31.117919 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:45:31.117919 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:45:31.117919 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:45:31.117919 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:45:31.117919 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:45:31.117919 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:45:31.117919 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:45:31.117919 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:45:31.117919 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:45:31.117919 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:45:31.117919 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:45:31.119716 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:45:31.119716 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:45:31.119716 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:45:31.119716 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 603/1215] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:45:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 269 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:45:31.123799 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed1fee4-1576-4f30-8ec3-c0fd02b9ffcf None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:31.198935 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:31.252801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed1fee4-1576-4f30-8ec3-c0fd02b9ffcf None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:31.304862 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb54c680-c52a-4eeb-aa10-75602d89afb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:31.305515 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb54c680-c52a-4eeb-aa10-75602d89afb6 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:31.305795 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb54c680-c52a-4eeb-aa10-75602d89afb6 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:31.306063 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb54c680-c52a-4eeb-aa10-75602d89afb6 None admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:31.306736 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb54c680-c52a-4eeb-aa10-75602d89afb6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:45:31.308065 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb54c680-c52a-4eeb-aa10-75602d89afb6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:31.308512 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb54c680-c52a-4eeb-aa10-75602d89afb6 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:31.324766 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 613/1216] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:45:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:45:31.332030 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cd82f69-563b-492b-b31c-8b1fa48e8e10 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:31.438656 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:31.483614 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cd82f69-563b-492b-b31c-8b1fa48e8e10 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:31.527848 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df9df9c0-5075-47f3-9288-81e19433ae20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:31.528510 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9df9c0-5075-47f3-9288-81e19433ae20 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:31.528787 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9df9c0-5075-47f3-9288-81e19433ae20 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:31.529057 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9df9c0-5075-47f3-9288-81e19433ae20 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:31.537395 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df9df9c0-5075-47f3-9288-81e19433ae20 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:31.539017 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df9df9c0-5075-47f3-9288-81e19433ae20 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:31.553074 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-df9df9c0-5075-47f3-9288-81e19433ae20 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:45:31.553074 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:45:31.553074 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:45:31.553074 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:45:31.553074 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:45:31.553074 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:45:31.553074 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:45:31.553074 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:45:31.553074 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:45:31.553074 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:45:31.553074 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:45:31.553074 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:45:31.553074 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 00:45:31.553074 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 00:45:31.553074 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 00:45:31.555839 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 00:45:31.555839 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:45:31.555839 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:45:31.555839 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:45:31.555839 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:45:31.555839 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:45:31.555839 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:45:31.555839 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:45:31.555839 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:45:31.555839 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:45:31.555839 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:45:31.555839 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:45:31.555839 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:45:31.555839 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:45:31.555839 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:45:31.555839 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:45:31.555839 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:45:31.558235 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 00:45:31.558235 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 00:45:31.558235 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 00:45:31.558235 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 00:45:31.558235 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 00:45:31.558235 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 00:45:31.558235 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:45:31.558235 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:45:31.558235 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 604/1217] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:45:31 2022] GET /identity/v3/projects/service => generated 88 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:45:31.564244 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb54c680-c52a-4eeb-aa10-75602d89afb6 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:31.730241 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb54c680-c52a-4eeb-aa10-75602d89afb6 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:31.765178 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:31.782422 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed00e3eb-0dbf-4a1a-a3e6-279e7a373337 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:31.783052 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed00e3eb-0dbf-4a1a-a3e6-279e7a373337 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:31.783332 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed00e3eb-0dbf-4a1a-a3e6-279e7a373337 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:31.783603 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed00e3eb-0dbf-4a1a-a3e6-279e7a373337 None admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:31.784301 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed00e3eb-0dbf-4a1a-a3e6-279e7a373337 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:45:31.784654 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed00e3eb-0dbf-4a1a-a3e6-279e7a373337 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:31.786272 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed00e3eb-0dbf-4a1a-a3e6-279e7a373337 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:31.800267 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 614/1218] 10.222.0.52 () {56 vars in 1148 bytes} [Sun Sep 25 00:45:31 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:45:31.806757 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df9df9c0-5075-47f3-9288-81e19433ae20 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:31.933136 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df9df9c0-5075-47f3-9288-81e19433ae20 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:31.977358 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:31.985882 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6304ddd5-7215-48bd-8cad-85e099f7d98a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:31.986755 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6304ddd5-7215-48bd-8cad-85e099f7d98a None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:31.987020 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6304ddd5-7215-48bd-8cad-85e099f7d98a None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:31.987292 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6304ddd5-7215-48bd-8cad-85e099f7d98a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:31.988023 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6304ddd5-7215-48bd-8cad-85e099f7d98a None admin] RBAC: Adding query filter params (role.id=a8e34e460e8a4ed7bcaa54da3c2d5a0c, scope.project.id=9db0fd07fc494203b2ada13f3cd046d3, user.id=11a80b55dcd444df901c8bf80f2a2c26) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:45:31.988325 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6304ddd5-7215-48bd-8cad-85e099f7d98a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:31.989823 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6304ddd5-7215-48bd-8cad-85e099f7d98a None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:31.995972 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 605/1219] 10.222.0.52 () {56 vars in 1374 bytes} [Sun Sep 25 00:45:31 2022] GET /identity/v3/role_assignments?user.id=11a80b55dcd444df901c8bf80f2a2c26&scope.project.id=9db0fd07fc494203b2ada13f3cd046d3&role.id=a8e34e460e8a4ed7bcaa54da3c2d5a0c => generated 259 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:45:34.402506 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e426c197-6ce3-4a79-81dd-631d3cafe416 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:34.403239 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e426c197-6ce3-4a79-81dd-631d3cafe416 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:34.403239 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e426c197-6ce3-4a79-81dd-631d3cafe416 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:34.404610 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 615/1220] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:45:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:45:34.412430 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e60387a-26e6-41bf-83fb-79c1a5a46e22 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:34.413331 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e60387a-26e6-41bf-83fb-79c1a5a46e22 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:34.413331 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e60387a-26e6-41bf-83fb-79c1a5a46e22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:34.438946 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-9e60387a-26e6-41bf-83fb-79c1a5a46e22 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:45:34.440390 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-9e60387a-26e6-41bf-83fb-79c1a5a46e22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:45:34.549686 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 606/1221] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:45:34 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:45:34.874824 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e426c197-6ce3-4a79-81dd-631d3cafe416 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:35.011533 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e426c197-6ce3-4a79-81dd-631d3cafe416 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:35.058551 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fcc36b4-7b6e-494d-820f-772948c27e41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:35.059530 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fcc36b4-7b6e-494d-820f-772948c27e41 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:35.060098 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fcc36b4-7b6e-494d-820f-772948c27e41 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:35.060642 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fcc36b4-7b6e-494d-820f-772948c27e41 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:35.080841 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fcc36b4-7b6e-494d-820f-772948c27e41 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:45:35.089601 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-5fcc36b4-7b6e-494d-820f-772948c27e41 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 25 00:45:35.089601 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:45:35.089601 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:45:35.089601 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:45:35.089601 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:45:35.089601 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:45:35.089601 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:45:35.089601 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:45:35.089601 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:45:35.089601 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:45:35.089601 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:45:35.089601 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:45:35.089601 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:45:35.089601 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:45:35.089601 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:45:35.089601 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:45:35.091774 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:45:35.091774 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:45:35.091774 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:45:35.091774 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:45:35.091774 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:45:35.091774 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:45:35.091774 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:45:35.091774 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:45:35.091774 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:45:35.091774 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:45:35.091774 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:45:35.091774 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:45:35.091774 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:45:35.091774 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:45:35.091774 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:45:35.091774 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:45:35.093205 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:45:35.093205 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:45:35.093205 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:45:35.093205 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:45:35.093205 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:45:35.093205 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:45:35.093205 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:45:35.093205 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:45:35.093205 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:45:35.093205 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:45:35.093205 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 25 00:45:35.093205 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:45:35.093205 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 616/1222] 10.222.0.52 () {56 vars in 1102 bytes} [Sun Sep 25 00:45:34 2022] GET /identity/v3/roles/service => generated 85 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:45:35.098208 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e60387a-26e6-41bf-83fb-79c1a5a46e22 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:35.214587 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e60387a-26e6-41bf-83fb-79c1a5a46e22 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:35.247609 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:35.266266 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f669910-7316-42f5-a3a1-fea8d10ead25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:35.267266 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f669910-7316-42f5-a3a1-fea8d10ead25 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:35.267726 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f669910-7316-42f5-a3a1-fea8d10ead25 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:35.268198 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f669910-7316-42f5-a3a1-fea8d10ead25 None admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:35.269429 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f669910-7316-42f5-a3a1-fea8d10ead25 None admin] RBAC: Adding query filter params (name=service) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:45:35.269925 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f669910-7316-42f5-a3a1-fea8d10ead25 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:35.272409 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f669910-7316-42f5-a3a1-fea8d10ead25 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:35.291413 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 607/1223] 10.222.0.52 () {56 vars in 1103 bytes} [Sun Sep 25 00:45:35 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:45:35.302023 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fcc36b4-7b6e-494d-820f-772948c27e41 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:35.376472 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:35.464083 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fcc36b4-7b6e-494d-820f-772948c27e41 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:35.522200 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-661d85f0-b2a9-45fb-bf93-9623a6ced88f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:35.523352 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661d85f0-b2a9-45fb-bf93-9623a6ced88f None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:35.523815 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661d85f0-b2a9-45fb-bf93-9623a6ced88f None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:35.524351 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661d85f0-b2a9-45fb-bf93-9623a6ced88f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:35.543363 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-661d85f0-b2a9-45fb-bf93-9623a6ced88f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:35.545082 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-661d85f0-b2a9-45fb-bf93-9623a6ced88f None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:35.563475 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-661d85f0-b2a9-45fb-bf93-9623a6ced88f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:45:35.563475 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:45:35.563475 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:45:35.563475 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:45:35.563475 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:45:35.563475 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:45:35.563475 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:45:35.563475 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:45:35.563475 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:45:35.563475 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:45:35.563475 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:45:35.563475 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:45:35.563475 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:45:35.563475 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:45:35.563475 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:45:35.563475 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:45:35.573842 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:45:35.573842 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:45:35.573842 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:45:35.573842 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:45:35.573842 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:45:35.573842 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:45:35.573842 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:45:35.573842 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:45:35.573842 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:45:35.573842 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:45:35.573842 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:45:35.573842 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:45:35.573842 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:45:35.573842 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:45:35.573842 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:45:35.573842 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:45:35.573842 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:45:35.576270 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:45:35.576270 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:45:35.576270 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:45:35.576270 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 617/1224] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:45:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 264 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:45:35.576831 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f669910-7316-42f5-a3a1-fea8d10ead25 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:35.720026 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f669910-7316-42f5-a3a1-fea8d10ead25 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:35.779618 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85d0054-0846-4bb3-8088-db3fe733f232 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:35.780358 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85d0054-0846-4bb3-8088-db3fe733f232 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:35.780887 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85d0054-0846-4bb3-8088-db3fe733f232 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:35.781274 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85d0054-0846-4bb3-8088-db3fe733f232 None admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:35.782045 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a85d0054-0846-4bb3-8088-db3fe733f232 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:45:35.782439 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a85d0054-0846-4bb3-8088-db3fe733f232 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:35.783962 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a85d0054-0846-4bb3-8088-db3fe733f232 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:35.797922 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 608/1225] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:45:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:45:35.804420 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-661d85f0-b2a9-45fb-bf93-9623a6ced88f None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:35.974691 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:35.986861 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-661d85f0-b2a9-45fb-bf93-9623a6ced88f None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:36.029881 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a8ec964-ade2-438a-a583-80e74753c377 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:36.030690 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8ec964-ade2-438a-a583-80e74753c377 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:36.031216 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8ec964-ade2-438a-a583-80e74753c377 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:36.031601 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8ec964-ade2-438a-a583-80e74753c377 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:36.043142 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a8ec964-ade2-438a-a583-80e74753c377 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:36.045996 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a8ec964-ade2-438a-a583-80e74753c377 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:36.062749 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-4a8ec964-ade2-438a-a583-80e74753c377 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 25 00:45:36.062749 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:45:36.062749 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:45:36.062749 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:45:36.062749 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:45:36.062749 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:45:36.062749 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:45:36.062749 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:45:36.062749 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:45:36.062749 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:45:36.062749 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:45:36.062749 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:45:36.062749 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 00:45:36.062749 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 00:45:36.062749 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 00:45:36.062749 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 00:45:36.073117 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:45:36.073117 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:45:36.073117 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 00:45:36.073117 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:45:36.073117 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 00:45:36.073117 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:45:36.073117 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:45:36.073117 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:45:36.073117 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:45:36.073117 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:45:36.073117 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:45:36.073117 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:45:36.073117 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:45:36.073117 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:45:36.073117 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:45:36.073117 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:45:36.073117 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:45:36.073117 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:45:36.075166 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:45:36.075166 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:45:36.075166 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 00:45:36.075166 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 00:45:36.075166 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 00:45:36.075166 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 00:45:36.075166 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 00:45:36.075166 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 00:45:36.075166 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 25 00:45:36.075166 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:45:36.075166 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 618/1226] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:45:35 2022] GET /identity/v3/users/placement => generated 87 bytes in 261 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:45:36.076813 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85d0054-0846-4bb3-8088-db3fe733f232 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:36.156814 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:36.240376 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85d0054-0846-4bb3-8088-db3fe733f232 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:36.309099 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-880614ce-09ff-4ea8-ae07-4d68f3afc73c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:36.309879 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880614ce-09ff-4ea8-ae07-4d68f3afc73c None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:36.311352 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880614ce-09ff-4ea8-ae07-4d68f3afc73c None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:36.311352 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880614ce-09ff-4ea8-ae07-4d68f3afc73c None admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:36.312838 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-880614ce-09ff-4ea8-ae07-4d68f3afc73c None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:45:36.313412 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-880614ce-09ff-4ea8-ae07-4d68f3afc73c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:36.316137 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-880614ce-09ff-4ea8-ae07-4d68f3afc73c None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:36.343599 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 609/1227] 10.222.0.52 () {56 vars in 1143 bytes} [Sun Sep 25 00:45:36 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 271 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:45:36.351003 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a8ec964-ade2-438a-a583-80e74753c377 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:36.556331 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a8ec964-ade2-438a-a583-80e74753c377 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:36.600688 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ef8ae56-2968-4c57-aefe-d43551d84090 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:36.601233 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef8ae56-2968-4c57-aefe-d43551d84090 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:36.601535 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef8ae56-2968-4c57-aefe-d43551d84090 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:36.601795 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef8ae56-2968-4c57-aefe-d43551d84090 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:36.615658 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ef8ae56-2968-4c57-aefe-d43551d84090 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:36.617111 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ef8ae56-2968-4c57-aefe-d43551d84090 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:36.635016 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-8ef8ae56-2968-4c57-aefe-d43551d84090 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:45:36.635016 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:45:36.635016 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:45:36.635016 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:45:36.635016 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:45:36.635016 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:45:36.635016 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:45:36.635016 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:45:36.635016 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:45:36.635016 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:45:36.635016 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:45:36.635016 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:45:36.635016 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:45:36.635016 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:45:36.635016 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:45:36.635016 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:45:36.637065 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:45:36.637065 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:45:36.637065 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:45:36.637065 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:45:36.637065 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:45:36.637065 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:45:36.637065 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:45:36.637065 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:45:36.637065 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:45:36.637065 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:45:36.637065 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:45:36.637065 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:45:36.637065 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:45:36.637065 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:45:36.637065 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:45:36.637065 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:45:36.637065 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:45:36.640890 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:45:36.640890 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:45:36.640890 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:45:36.640890 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 619/1228] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:45:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 287 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:45:36.644997 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-880614ce-09ff-4ea8-ae07-4d68f3afc73c None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:36.790461 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-880614ce-09ff-4ea8-ae07-4d68f3afc73c None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:36.793931 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:36.848113 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515746fe-132d-4548-8862-b49f1157d2ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:36.848791 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515746fe-132d-4548-8862-b49f1157d2ee None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:36.849168 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515746fe-132d-4548-8862-b49f1157d2ee None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:36.849478 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515746fe-132d-4548-8862-b49f1157d2ee None admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:36.850128 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-515746fe-132d-4548-8862-b49f1157d2ee None admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:45:36.850489 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-515746fe-132d-4548-8862-b49f1157d2ee None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:36.852084 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-515746fe-132d-4548-8862-b49f1157d2ee None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:36.870871 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 610/1229] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:45:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:45:36.879871 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ef8ae56-2968-4c57-aefe-d43551d84090 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:36.895654 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:37.045917 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ef8ae56-2968-4c57-aefe-d43551d84090 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:37.109874 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9c4dbf-c0cb-4a00-ab31-2d208f0b1632 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:37.110817 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9c4dbf-c0cb-4a00-ab31-2d208f0b1632 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:37.111553 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9c4dbf-c0cb-4a00-ab31-2d208f0b1632 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:37.112021 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9c4dbf-c0cb-4a00-ab31-2d208f0b1632 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:37.125069 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b9c4dbf-c0cb-4a00-ab31-2d208f0b1632 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:37.127870 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b9c4dbf-c0cb-4a00-ab31-2d208f0b1632 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:37.150510 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-5b9c4dbf-c0cb-4a00-ab31-2d208f0b1632 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:45:37.150510 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:45:37.150510 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:45:37.150510 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:45:37.150510 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:45:37.150510 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:45:37.150510 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:45:37.150510 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:45:37.150510 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:45:37.150510 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:45:37.150510 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:45:37.150510 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:45:37.150510 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 00:45:37.150510 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 00:45:37.150510 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 00:45:37.152367 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 00:45:37.152367 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:45:37.152367 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:45:37.152367 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:45:37.152367 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:45:37.152367 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:45:37.152367 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:45:37.152367 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:45:37.152367 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:45:37.152367 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:45:37.152367 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:45:37.152367 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:45:37.152367 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:45:37.152367 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:45:37.152367 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:45:37.152367 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:45:37.152367 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:45:37.154693 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 00:45:37.154693 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 00:45:37.154693 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 00:45:37.154693 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 00:45:37.154693 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 00:45:37.154693 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 00:45:37.154693 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:45:37.154693 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:45:37.154693 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 620/1230] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:45:36 2022] GET /identity/v3/projects/service => generated 88 bytes in 273 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:45:37.163188 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515746fe-132d-4548-8862-b49f1157d2ee None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:37.320266 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515746fe-132d-4548-8862-b49f1157d2ee None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:37.366249 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd18a13a-b19a-4bc1-9acd-24498bc03c9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:37.366788 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd18a13a-b19a-4bc1-9acd-24498bc03c9e None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:37.367061 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd18a13a-b19a-4bc1-9acd-24498bc03c9e None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:37.367325 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd18a13a-b19a-4bc1-9acd-24498bc03c9e None admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:37.368034 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd18a13a-b19a-4bc1-9acd-24498bc03c9e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:45:37.368336 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd18a13a-b19a-4bc1-9acd-24498bc03c9e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:37.369839 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd18a13a-b19a-4bc1-9acd-24498bc03c9e None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:37.386253 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 611/1231] 10.222.0.52 () {56 vars in 1148 bytes} [Sun Sep 25 00:45:37 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:45:37.394007 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9c4dbf-c0cb-4a00-ab31-2d208f0b1632 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:37.485123 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:37.549282 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9c4dbf-c0cb-4a00-ab31-2d208f0b1632 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:37.599139 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de67d904-9d0e-4547-99de-c11ccc63154b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:37.600050 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de67d904-9d0e-4547-99de-c11ccc63154b None admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:37.600402 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de67d904-9d0e-4547-99de-c11ccc63154b None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:37.600836 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de67d904-9d0e-4547-99de-c11ccc63154b None admin] PATH_INFO: `/v3/projects/9db0fd07fc494203b2ada13f3cd046d3/users/11a80b55dcd444df901c8bf80f2a2c26/roles/a8e34e460e8a4ed7bcaa54da3c2d5a0c` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:37.624528 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de67d904-9d0e-4547-99de-c11ccc63154b None admin] RBAC: Authorizing `identity:create_grant(project_id=9db0fd07fc494203b2ada13f3cd046d3, user_id=11a80b55dcd444df901c8bf80f2a2c26, role_id=a8e34e460e8a4ed7bcaa54da3c2d5a0c)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:37.625945 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de67d904-9d0e-4547-99de-c11ccc63154b None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:37.649388 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 621/1232] 10.222.0.52 () {58 vars in 1439 bytes} [Sun Sep 25 00:45:37 2022] PUT /identity/v3/projects/9db0fd07fc494203b2ada13f3cd046d3/users/11a80b55dcd444df901c8bf80f2a2c26/roles/a8e34e460e8a4ed7bcaa54da3c2d5a0c => generated 0 bytes in 250 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:45:40.015520 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32cf347b-abc2-4a57-a714-1df857ee6546 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:40.016138 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32cf347b-abc2-4a57-a714-1df857ee6546 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:40.016138 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32cf347b-abc2-4a57-a714-1df857ee6546 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:40.017653 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 612/1233] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:45:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:45:40.023569 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-058ec62a-1b2c-4f28-a287-b5e14aa5fd3e None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:40.024205 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-058ec62a-1b2c-4f28-a287-b5e14aa5fd3e None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:40.024205 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-058ec62a-1b2c-4f28-a287-b5e14aa5fd3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:40.044752 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-058ec62a-1b2c-4f28-a287-b5e14aa5fd3e None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:45:40.046452 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-058ec62a-1b2c-4f28-a287-b5e14aa5fd3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:45:40.140026 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 622/1234] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:45:40 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:45:40.415651 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32cf347b-abc2-4a57-a714-1df857ee6546 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:40.444566 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:40.562090 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32cf347b-abc2-4a57-a714-1df857ee6546 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:40.622222 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe2365a-b107-49cc-a545-632082491054 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:40.624504 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe2365a-b107-49cc-a545-632082491054 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:40.624954 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe2365a-b107-49cc-a545-632082491054 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:40.625437 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe2365a-b107-49cc-a545-632082491054 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:40.642213 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfe2365a-b107-49cc-a545-632082491054 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:45:40.648086 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-cfe2365a-b107-49cc-a545-632082491054 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 25 00:45:40.648086 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:45:40.648086 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:45:40.648086 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:45:40.648086 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:45:40.648086 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:45:40.648086 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:45:40.648086 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:45:40.648086 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:45:40.648086 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:45:40.648086 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:45:40.648086 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:45:40.648086 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:45:40.648086 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:45:40.648086 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:45:40.648086 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:45:40.650167 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:45:40.650167 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:45:40.650167 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:45:40.650167 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:45:40.650167 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:45:40.650167 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:45:40.650167 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:45:40.650167 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:45:40.650167 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:45:40.650167 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:45:40.650167 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:45:40.650167 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:45:40.650167 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:45:40.650167 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:45:40.650167 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:45:40.650167 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:45:40.652036 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:45:40.652036 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:45:40.652036 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:45:40.652036 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:45:40.652036 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:45:40.652036 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:45:40.652036 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:45:40.652036 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:45:40.652036 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:45:40.652036 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:45:40.652036 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 25 00:45:40.652036 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:45:40.652036 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 613/1235] 10.222.0.52 () {56 vars in 1102 bytes} [Sun Sep 25 00:45:40 2022] GET /identity/v3/roles/service => generated 85 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:45:40.659355 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-058ec62a-1b2c-4f28-a287-b5e14aa5fd3e None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:40.719657 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:40.805707 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-058ec62a-1b2c-4f28-a287-b5e14aa5fd3e None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:40.859751 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-182bd3c5-5cec-45d5-a6e2-e6555f028176 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:40.860365 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182bd3c5-5cec-45d5-a6e2-e6555f028176 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:40.860638 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182bd3c5-5cec-45d5-a6e2-e6555f028176 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:40.861177 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182bd3c5-5cec-45d5-a6e2-e6555f028176 None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:40.861862 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-182bd3c5-5cec-45d5-a6e2-e6555f028176 None admin] RBAC: Adding query filter params (name=service) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:45:40.862161 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-182bd3c5-5cec-45d5-a6e2-e6555f028176 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:40.863537 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-182bd3c5-5cec-45d5-a6e2-e6555f028176 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:40.874365 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 623/1236] 10.222.0.52 () {56 vars in 1103 bytes} [Sun Sep 25 00:45:40 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:45:40.883192 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe2365a-b107-49cc-a545-632082491054 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:41.012352 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:41.035167 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe2365a-b107-49cc-a545-632082491054 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:41.083042 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26d3e2d-39e1-4660-b6a5-d0d61bc47d6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:41.084101 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26d3e2d-39e1-4660-b6a5-d0d61bc47d6c None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:41.084570 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26d3e2d-39e1-4660-b6a5-d0d61bc47d6c None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:41.085008 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26d3e2d-39e1-4660-b6a5-d0d61bc47d6c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:41.104208 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f26d3e2d-39e1-4660-b6a5-d0d61bc47d6c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:41.107391 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f26d3e2d-39e1-4660-b6a5-d0d61bc47d6c None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:41.137072 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-f26d3e2d-39e1-4660-b6a5-d0d61bc47d6c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:45:41.137072 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:45:41.137072 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:45:41.137072 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:45:41.137072 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:45:41.137072 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:45:41.137072 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:45:41.137072 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:45:41.137072 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:45:41.137072 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:45:41.137072 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:45:41.137072 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:45:41.137072 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:45:41.137072 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:45:41.137072 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:45:41.137072 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:45:41.139539 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:45:41.139539 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:45:41.139539 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:45:41.139539 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:45:41.139539 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:45:41.139539 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:45:41.139539 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:45:41.139539 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:45:41.139539 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:45:41.139539 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:45:41.139539 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:45:41.139539 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:45:41.139539 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:45:41.139539 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:45:41.139539 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:45:41.139539 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:45:41.139539 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:45:41.140849 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:45:41.140849 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:45:41.140849 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:45:41.140849 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 614/1237] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:45:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 258 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:45:41.149720 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-182bd3c5-5cec-45d5-a6e2-e6555f028176 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:41.323624 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-182bd3c5-5cec-45d5-a6e2-e6555f028176 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:41.346652 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:41.388758 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2a61c8-c810-4fda-9fa8-669eeaf5e722 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:41.389576 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2a61c8-c810-4fda-9fa8-669eeaf5e722 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:41.389893 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2a61c8-c810-4fda-9fa8-669eeaf5e722 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:41.390180 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2a61c8-c810-4fda-9fa8-669eeaf5e722 None admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:41.390967 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f2a61c8-c810-4fda-9fa8-669eeaf5e722 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:45:41.391306 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f2a61c8-c810-4fda-9fa8-669eeaf5e722 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:41.392868 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f2a61c8-c810-4fda-9fa8-669eeaf5e722 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:41.408867 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 624/1238] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:45:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:45:41.414933 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26d3e2d-39e1-4660-b6a5-d0d61bc47d6c None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:41.578360 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26d3e2d-39e1-4660-b6a5-d0d61bc47d6c None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:41.631641 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a845b964-a241-46b3-b27f-a9a251ae4c7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:41.632593 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a845b964-a241-46b3-b27f-a9a251ae4c7e None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:41.632989 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a845b964-a241-46b3-b27f-a9a251ae4c7e None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:41.633369 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a845b964-a241-46b3-b27f-a9a251ae4c7e None admin] PATH_INFO: `/v3/users/placement` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:41.638652 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:41.649920 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a845b964-a241-46b3-b27f-a9a251ae4c7e None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:41.653129 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a845b964-a241-46b3-b27f-a9a251ae4c7e None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:41.672688 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-a845b964-a241-46b3-b27f-a9a251ae4c7e None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 25 00:45:41.672688 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:45:41.672688 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:45:41.672688 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:45:41.672688 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:45:41.672688 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:45:41.672688 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:45:41.672688 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:45:41.672688 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:45:41.672688 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:45:41.672688 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:45:41.672688 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:45:41.672688 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 00:45:41.672688 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 00:45:41.672688 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 00:45:41.672688 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 00:45:41.674307 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:45:41.674307 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:45:41.674307 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 00:45:41.674307 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:45:41.674307 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 00:45:41.674307 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:45:41.674307 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:45:41.674307 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:45:41.674307 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:45:41.674307 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:45:41.674307 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:45:41.674307 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:45:41.674307 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:45:41.674307 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:45:41.674307 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:45:41.674307 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:45:41.674307 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:45:41.674307 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:45:41.675975 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:45:41.675975 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:45:41.675975 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 00:45:41.675975 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 00:45:41.675975 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 00:45:41.675975 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 00:45:41.675975 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 00:45:41.675975 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 00:45:41.675975 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 25 00:45:41.675975 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:45:41.675975 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 615/1239] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:45:41 2022] GET /identity/v3/users/placement => generated 87 bytes in 260 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:45:41.680393 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2a61c8-c810-4fda-9fa8-669eeaf5e722 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:41.839150 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2a61c8-c810-4fda-9fa8-669eeaf5e722 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:41.893482 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c939c6-3e88-4174-9b53-94d825b679d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:41.894435 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c939c6-3e88-4174-9b53-94d825b679d4 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:41.895040 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c939c6-3e88-4174-9b53-94d825b679d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:41.895335 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c939c6-3e88-4174-9b53-94d825b679d4 None admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:41.896296 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c939c6-3e88-4174-9b53-94d825b679d4 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:45:41.896664 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c939c6-3e88-4174-9b53-94d825b679d4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:41.898401 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c939c6-3e88-4174-9b53-94d825b679d4 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:41.923592 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 625/1240] 10.222.0.52 () {56 vars in 1143 bytes} [Sun Sep 25 00:45:41 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:45:41.929653 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a845b964-a241-46b3-b27f-a9a251ae4c7e None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:42.071274 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a845b964-a241-46b3-b27f-a9a251ae4c7e None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:42.125988 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76ad3b21-f22c-4ef0-b8e5-3fe89ca387dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:42.126618 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ad3b21-f22c-4ef0-b8e5-3fe89ca387dc None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:42.126996 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ad3b21-f22c-4ef0-b8e5-3fe89ca387dc None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:42.127623 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ad3b21-f22c-4ef0-b8e5-3fe89ca387dc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:42.232293 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:42.244734 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76ad3b21-f22c-4ef0-b8e5-3fe89ca387dc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:42.246816 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76ad3b21-f22c-4ef0-b8e5-3fe89ca387dc None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:42.394164 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 616/1241] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:45:41 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 465 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 25 00:45:42.400651 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c939c6-3e88-4174-9b53-94d825b679d4 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:42.433733 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:42.537143 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c939c6-3e88-4174-9b53-94d825b679d4 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:42.587540 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-180fa236-c8bf-4d33-b3cf-117e70ffdd96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:42.588622 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180fa236-c8bf-4d33-b3cf-117e70ffdd96 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:42.589075 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180fa236-c8bf-4d33-b3cf-117e70ffdd96 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:42.589568 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180fa236-c8bf-4d33-b3cf-117e70ffdd96 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:42.604337 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-180fa236-c8bf-4d33-b3cf-117e70ffdd96 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:42.606574 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-180fa236-c8bf-4d33-b3cf-117e70ffdd96 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:42.626075 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-180fa236-c8bf-4d33-b3cf-117e70ffdd96 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:45:42.626075 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:45:42.626075 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:45:42.626075 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:45:42.626075 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:45:42.626075 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:45:42.626075 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:45:42.626075 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:45:42.626075 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:45:42.626075 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:45:42.626075 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:45:42.626075 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:45:42.626075 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:45:42.626075 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:45:42.626075 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:45:42.626075 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:45:42.627918 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:45:42.627918 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:45:42.627918 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:45:42.627918 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:45:42.627918 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:45:42.627918 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:45:42.627918 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:45:42.627918 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:45:42.627918 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:45:42.627918 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:45:42.627918 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:45:42.627918 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:45:42.627918 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:45:42.627918 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:45:42.627918 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:45:42.627918 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:45:42.627918 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:45:42.629118 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:45:42.629118 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:45:42.629118 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:45:42.629118 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 626/1242] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:45:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:45:42.637221 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76ad3b21-f22c-4ef0-b8e5-3fe89ca387dc None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:42.796436 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:42.806070 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76ad3b21-f22c-4ef0-b8e5-3fe89ca387dc None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:42.863636 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeed9f96-a81e-48f4-885a-8c4363fc783c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:42.864350 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeed9f96-a81e-48f4-885a-8c4363fc783c None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:42.864679 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeed9f96-a81e-48f4-885a-8c4363fc783c None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:42.865039 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeed9f96-a81e-48f4-885a-8c4363fc783c None admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:42.866074 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeed9f96-a81e-48f4-885a-8c4363fc783c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:45:42.866458 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeed9f96-a81e-48f4-885a-8c4363fc783c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:42.868412 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeed9f96-a81e-48f4-885a-8c4363fc783c None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:42.884871 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 617/1243] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:45:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:45:42.891696 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-180fa236-c8bf-4d33-b3cf-117e70ffdd96 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:43.030995 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:43.069632 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-180fa236-c8bf-4d33-b3cf-117e70ffdd96 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:43.114715 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d316ec6f-26d5-4eab-81e4-9169b1a4c458 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:43.115357 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d316ec6f-26d5-4eab-81e4-9169b1a4c458 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:43.115634 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d316ec6f-26d5-4eab-81e4-9169b1a4c458 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:43.115905 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d316ec6f-26d5-4eab-81e4-9169b1a4c458 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:43.124907 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d316ec6f-26d5-4eab-81e4-9169b1a4c458 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:43.126423 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d316ec6f-26d5-4eab-81e4-9169b1a4c458 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:43.148730 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-d316ec6f-26d5-4eab-81e4-9169b1a4c458 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:45:43.148730 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:45:43.148730 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:45:43.148730 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:45:43.148730 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:45:43.148730 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:45:43.148730 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:45:43.148730 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:45:43.148730 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:45:43.148730 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:45:43.148730 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:45:43.148730 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:45:43.148730 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 00:45:43.148730 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 00:45:43.148730 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 00:45:43.151154 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 00:45:43.151154 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:45:43.151154 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:45:43.151154 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:45:43.151154 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:45:43.151154 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:45:43.151154 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:45:43.151154 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:45:43.151154 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:45:43.151154 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:45:43.151154 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:45:43.151154 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:45:43.151154 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:45:43.151154 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:45:43.151154 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:45:43.151154 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:45:43.151154 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:45:43.153112 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 00:45:43.153112 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 00:45:43.153112 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 00:45:43.153112 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 00:45:43.153112 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 00:45:43.153112 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 00:45:43.153112 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:45:43.153112 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:45:43.153112 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 627/1244] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:45:42 2022] GET /identity/v3/projects/service => generated 88 bytes in 259 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:45:43.156951 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeed9f96-a81e-48f4-885a-8c4363fc783c None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:43.302157 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeed9f96-a81e-48f4-885a-8c4363fc783c None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:43.350370 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d4c9e69-bdbb-4730-8ffd-689710f54ff4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:43.351084 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d4c9e69-bdbb-4730-8ffd-689710f54ff4 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:43.351446 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d4c9e69-bdbb-4730-8ffd-689710f54ff4 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:43.351803 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d4c9e69-bdbb-4730-8ffd-689710f54ff4 None admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:43.352624 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d4c9e69-bdbb-4730-8ffd-689710f54ff4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:45:43.353025 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d4c9e69-bdbb-4730-8ffd-689710f54ff4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:43.355267 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d4c9e69-bdbb-4730-8ffd-689710f54ff4 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:43.377123 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 618/1245] 10.222.0.52 () {56 vars in 1148 bytes} [Sun Sep 25 00:45:43 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:45:43.384307 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d316ec6f-26d5-4eab-81e4-9169b1a4c458 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:43.511854 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d316ec6f-26d5-4eab-81e4-9169b1a4c458 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:43.547855 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1040fdda-b4ec-4939-9129-0a4d6c9c617e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:43.548465 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1040fdda-b4ec-4939-9129-0a4d6c9c617e None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:43.548741 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1040fdda-b4ec-4939-9129-0a4d6c9c617e None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:43.549014 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1040fdda-b4ec-4939-9129-0a4d6c9c617e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:43.549926 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1040fdda-b4ec-4939-9129-0a4d6c9c617e None admin] RBAC: Adding query filter params (role.id=a8e34e460e8a4ed7bcaa54da3c2d5a0c, scope.project.id=9db0fd07fc494203b2ada13f3cd046d3, user.id=11a80b55dcd444df901c8bf80f2a2c26) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:45:43.550241 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1040fdda-b4ec-4939-9129-0a4d6c9c617e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:43.551687 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1040fdda-b4ec-4939-9129-0a4d6c9c617e None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:43.560687 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 628/1246] 10.222.0.52 () {56 vars in 1374 bytes} [Sun Sep 25 00:45:43 2022] GET /identity/v3/role_assignments?user.id=11a80b55dcd444df901c8bf80f2a2c26&scope.project.id=9db0fd07fc494203b2ada13f3cd046d3&role.id=a8e34e460e8a4ed7bcaa54da3c2d5a0c => generated 608 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:45:46.266862 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fea286-b406-4881-8e2f-706d0f888448 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:46.267695 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fea286-b406-4881-8e2f-706d0f888448 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:46.267695 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fea286-b406-4881-8e2f-706d0f888448 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:46.269074 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 619/1247] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:45:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 00:45:46.276321 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8395c1c6-c129-4d57-b914-512c4998394f None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:46.277292 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8395c1c6-c129-4d57-b914-512c4998394f None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:46.277292 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8395c1c6-c129-4d57-b914-512c4998394f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:46.285542 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:46.300612 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-8395c1c6-c129-4d57-b914-512c4998394f None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:45:46.301871 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-8395c1c6-c129-4d57-b914-512c4998394f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:45:46.407271 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 629/1248] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:45:46 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:45:46.706574 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21fea286-b406-4881-8e2f-706d0f888448 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:46.729006 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:46.866724 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21fea286-b406-4881-8e2f-706d0f888448 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:46.931579 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3197514d-e216-4ffb-b35a-c96f38a4abc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:46.932507 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3197514d-e216-4ffb-b35a-c96f38a4abc4 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:46.932858 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3197514d-e216-4ffb-b35a-c96f38a4abc4 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:46.933166 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3197514d-e216-4ffb-b35a-c96f38a4abc4 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:46.950089 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3197514d-e216-4ffb-b35a-c96f38a4abc4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:45:46.957728 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-3197514d-e216-4ffb-b35a-c96f38a4abc4 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:45:46.957728 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:45:46.957728 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:45:46.957728 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:45:46.957728 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:45:46.957728 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:45:46.957728 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:45:46.957728 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:45:46.957728 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:45:46.957728 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:45:46.957728 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:45:46.957728 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:45:46.957728 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:45:46.957728 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:45:46.957728 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:45:46.957728 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:45:46.959453 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:45:46.959453 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:45:46.959453 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:45:46.959453 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:45:46.959453 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:45:46.959453 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:45:46.959453 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:45:46.959453 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:45:46.959453 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:45:46.959453 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:45:46.959453 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:45:46.959453 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:45:46.959453 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:45:46.959453 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:45:46.959453 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:45:46.959453 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:45:46.960656 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:45:46.960656 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:45:46.960656 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:45:46.960656 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:45:46.960656 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:45:46.960656 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:45:46.960656 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:45:46.960656 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:45:46.960656 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:45:46.960656 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:45:46.960656 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:45:46.960656 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:45:46.960656 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 620/1249] 10.222.0.52 () {56 vars in 1096 bytes} [Sun Sep 25 00:45:46 2022] GET /identity/v3/roles/admin => generated 83 bytes in 254 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:45:46.969207 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8395c1c6-c129-4d57-b914-512c4998394f None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:47.130955 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8395c1c6-c129-4d57-b914-512c4998394f None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:47.176235 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:47.183217 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0565dbba-fd56-4ed6-aaf4-4fe0cc5a7d6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:47.183911 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0565dbba-fd56-4ed6-aaf4-4fe0cc5a7d6c None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:47.184129 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0565dbba-fd56-4ed6-aaf4-4fe0cc5a7d6c None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:47.184359 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0565dbba-fd56-4ed6-aaf4-4fe0cc5a7d6c None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:47.185300 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0565dbba-fd56-4ed6-aaf4-4fe0cc5a7d6c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:45:47.185934 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0565dbba-fd56-4ed6-aaf4-4fe0cc5a7d6c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:47.187357 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0565dbba-fd56-4ed6-aaf4-4fe0cc5a7d6c None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:47.196921 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 630/1250] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:45:46 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:45:47.205118 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3197514d-e216-4ffb-b35a-c96f38a4abc4 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:47.330006 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:47.374164 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3197514d-e216-4ffb-b35a-c96f38a4abc4 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:47.434051 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-218cf1ba-bef6-4ffc-9ead-121d2b663792 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:47.435562 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-218cf1ba-bef6-4ffc-9ead-121d2b663792 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:47.436254 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-218cf1ba-bef6-4ffc-9ead-121d2b663792 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:47.436644 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-218cf1ba-bef6-4ffc-9ead-121d2b663792 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:47.453988 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-218cf1ba-bef6-4ffc-9ead-121d2b663792 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:47.455660 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-218cf1ba-bef6-4ffc-9ead-121d2b663792 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:47.476680 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-218cf1ba-bef6-4ffc-9ead-121d2b663792 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:45:47.476680 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:45:47.476680 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:45:47.476680 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:45:47.476680 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:45:47.476680 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:45:47.476680 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:45:47.476680 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:45:47.476680 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:45:47.476680 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:45:47.476680 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:45:47.476680 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:45:47.476680 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:45:47.476680 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:45:47.476680 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:45:47.476680 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:45:47.478536 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:45:47.478536 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:45:47.478536 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:45:47.478536 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:45:47.478536 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:45:47.478536 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:45:47.478536 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:45:47.478536 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:45:47.478536 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:45:47.478536 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:45:47.478536 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:45:47.478536 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:45:47.478536 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:45:47.478536 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:45:47.478536 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:45:47.478536 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:45:47.478536 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:45:47.479667 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:45:47.479667 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:45:47.479667 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:45:47.479667 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 621/1251] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:45:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 275 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:45:47.487897 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0565dbba-fd56-4ed6-aaf4-4fe0cc5a7d6c None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:47.651023 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0565dbba-fd56-4ed6-aaf4-4fe0cc5a7d6c None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:47.701583 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce807c4b-2837-419f-a8a9-f5b213f5fb66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:47.703280 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce807c4b-2837-419f-a8a9-f5b213f5fb66 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:47.704175 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce807c4b-2837-419f-a8a9-f5b213f5fb66 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:47.704634 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce807c4b-2837-419f-a8a9-f5b213f5fb66 None admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:47.706170 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce807c4b-2837-419f-a8a9-f5b213f5fb66 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:45:47.706900 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce807c4b-2837-419f-a8a9-f5b213f5fb66 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:47.709753 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce807c4b-2837-419f-a8a9-f5b213f5fb66 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:47.728685 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 631/1252] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:45:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:45:47.735176 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-218cf1ba-bef6-4ffc-9ead-121d2b663792 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:47.895603 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-218cf1ba-bef6-4ffc-9ead-121d2b663792 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:47.942646 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:47.955286 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e7d75a-623d-4b87-b5ec-bcac48189f9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:47.955996 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e7d75a-623d-4b87-b5ec-bcac48189f9b None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:47.956308 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e7d75a-623d-4b87-b5ec-bcac48189f9b None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:47.956617 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e7d75a-623d-4b87-b5ec-bcac48189f9b None admin] PATH_INFO: `/v3/users/placement` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:47.967276 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5e7d75a-623d-4b87-b5ec-bcac48189f9b None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:47.968801 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5e7d75a-623d-4b87-b5ec-bcac48189f9b None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:47.988064 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-c5e7d75a-623d-4b87-b5ec-bcac48189f9b None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 25 00:45:47.988064 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:45:47.988064 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:45:47.988064 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:45:47.988064 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:45:47.988064 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:45:47.988064 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:45:47.988064 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:45:47.988064 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:45:47.988064 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:45:47.988064 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:45:47.988064 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:45:47.988064 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 00:45:47.988064 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 00:45:47.988064 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 00:45:47.988064 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 00:45:47.990669 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:45:47.990669 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:45:47.990669 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 00:45:47.990669 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:45:47.990669 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 00:45:47.990669 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:45:47.990669 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:45:47.990669 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:45:47.990669 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:45:47.990669 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:45:47.990669 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:45:47.990669 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:45:47.990669 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:45:47.990669 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:45:47.990669 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:45:47.990669 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:45:47.990669 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:45:47.990669 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:45:47.997360 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:45:47.997360 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:45:47.997360 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 00:45:47.997360 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 00:45:47.997360 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 00:45:47.997360 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 00:45:47.997360 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 00:45:47.997360 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 00:45:47.997360 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 25 00:45:47.997360 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:45:47.997360 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 622/1253] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:45:47 2022] GET /identity/v3/users/placement => generated 87 bytes in 256 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:45:47.999366 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce807c4b-2837-419f-a8a9-f5b213f5fb66 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:48.136774 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:48.211514 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce807c4b-2837-419f-a8a9-f5b213f5fb66 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:48.252349 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf73193-a5a0-4ed7-a2d5-ad9983461936 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:48.253145 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf73193-a5a0-4ed7-a2d5-ad9983461936 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:48.253624 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf73193-a5a0-4ed7-a2d5-ad9983461936 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:48.254084 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf73193-a5a0-4ed7-a2d5-ad9983461936 None admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:48.255266 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adf73193-a5a0-4ed7-a2d5-ad9983461936 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:45:48.255579 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adf73193-a5a0-4ed7-a2d5-ad9983461936 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:48.257322 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adf73193-a5a0-4ed7-a2d5-ad9983461936 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:48.284796 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 632/1254] 10.222.0.52 () {56 vars in 1143 bytes} [Sun Sep 25 00:45:47 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 288 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:45:48.292180 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e7d75a-623d-4b87-b5ec-bcac48189f9b None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:48.421662 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e7d75a-623d-4b87-b5ec-bcac48189f9b None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:48.477426 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6670e54f-1c67-4771-9c5e-2a519ed425a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:48.478498 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6670e54f-1c67-4771-9c5e-2a519ed425a8 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:48.479005 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6670e54f-1c67-4771-9c5e-2a519ed425a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:48.479504 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6670e54f-1c67-4771-9c5e-2a519ed425a8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:48.585949 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:48.603544 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6670e54f-1c67-4771-9c5e-2a519ed425a8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:48.605004 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6670e54f-1c67-4771-9c5e-2a519ed425a8 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:48.770753 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 623/1255] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:45:48 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 480 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 25 00:45:48.778374 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf73193-a5a0-4ed7-a2d5-ad9983461936 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:48.916458 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf73193-a5a0-4ed7-a2d5-ad9983461936 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:48.940589 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:48.975180 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-318c676f-c027-4c25-b3e9-e94d2197e211 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:48.975945 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318c676f-c027-4c25-b3e9-e94d2197e211 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:48.976174 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318c676f-c027-4c25-b3e9-e94d2197e211 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:48.976447 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318c676f-c027-4c25-b3e9-e94d2197e211 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:48.989340 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-318c676f-c027-4c25-b3e9-e94d2197e211 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:48.990742 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-318c676f-c027-4c25-b3e9-e94d2197e211 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:49.008396 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-318c676f-c027-4c25-b3e9-e94d2197e211 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:45:49.008396 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:45:49.008396 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:45:49.008396 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:45:49.008396 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:45:49.008396 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:45:49.008396 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:45:49.008396 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:45:49.008396 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:45:49.008396 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:45:49.008396 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:45:49.008396 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:45:49.008396 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:45:49.008396 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:45:49.008396 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:45:49.008396 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:45:49.010536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:45:49.010536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:45:49.010536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:45:49.010536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:45:49.010536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:45:49.010536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:45:49.010536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:45:49.010536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:45:49.010536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:45:49.010536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:45:49.010536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:45:49.010536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:45:49.010536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:45:49.010536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:45:49.010536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:45:49.010536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:45:49.010536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:45:49.012563 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:45:49.012563 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:45:49.012563 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:45:49.012563 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 633/1256] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:45:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:45:49.019343 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6670e54f-1c67-4771-9c5e-2a519ed425a8 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:49.165378 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:49.181637 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6670e54f-1c67-4771-9c5e-2a519ed425a8 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:49.225712 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2484096a-9ebf-455d-99ef-ad24b1eb1a1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:49.226401 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2484096a-9ebf-455d-99ef-ad24b1eb1a1b None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:49.226674 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2484096a-9ebf-455d-99ef-ad24b1eb1a1b None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:49.226940 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2484096a-9ebf-455d-99ef-ad24b1eb1a1b None admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:49.227656 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2484096a-9ebf-455d-99ef-ad24b1eb1a1b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:45:49.227950 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2484096a-9ebf-455d-99ef-ad24b1eb1a1b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:49.229447 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2484096a-9ebf-455d-99ef-ad24b1eb1a1b None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:49.242839 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 624/1257] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:45:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:45:49.252078 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-318c676f-c027-4c25-b3e9-e94d2197e211 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:49.380483 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-318c676f-c027-4c25-b3e9-e94d2197e211 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:49.423343 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703467a5-4dcf-40c4-8009-91a0926bc122 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:49.423951 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703467a5-4dcf-40c4-8009-91a0926bc122 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:49.424136 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703467a5-4dcf-40c4-8009-91a0926bc122 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:49.424490 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703467a5-4dcf-40c4-8009-91a0926bc122 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:49.434233 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-703467a5-4dcf-40c4-8009-91a0926bc122 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:49.436477 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-703467a5-4dcf-40c4-8009-91a0926bc122 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:49.453171 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-703467a5-4dcf-40c4-8009-91a0926bc122 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:45:49.453171 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:45:49.453171 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:45:49.453171 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:45:49.453171 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:45:49.453171 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:45:49.453171 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:45:49.453171 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:45:49.453171 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:45:49.453171 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:45:49.453171 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:45:49.453171 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:45:49.453171 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 00:45:49.453171 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 00:45:49.453171 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 00:45:49.455306 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 00:45:49.455306 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:45:49.455306 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:45:49.455306 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:45:49.455306 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:45:49.455306 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:45:49.455306 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:45:49.455306 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:45:49.455306 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:45:49.455306 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:45:49.455306 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:45:49.455306 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:45:49.455306 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:45:49.455306 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:45:49.455306 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:45:49.455306 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:45:49.455306 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:45:49.457796 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 00:45:49.457796 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 00:45:49.457796 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 00:45:49.457796 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 00:45:49.457796 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 00:45:49.457796 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 00:45:49.457796 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:45:49.457796 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:45:49.457796 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 634/1258] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:45:49 2022] GET /identity/v3/projects/service => generated 88 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:45:49.461888 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2484096a-9ebf-455d-99ef-ad24b1eb1a1b None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:49.634122 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2484096a-9ebf-455d-99ef-ad24b1eb1a1b None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:49.687652 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f850fd-be72-4b83-8036-30b31f441cb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:49.688252 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f850fd-be72-4b83-8036-30b31f441cb2 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:49.688568 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f850fd-be72-4b83-8036-30b31f441cb2 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:49.688881 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f850fd-be72-4b83-8036-30b31f441cb2 None admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:49.689874 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98f850fd-be72-4b83-8036-30b31f441cb2 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:45:49.690176 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98f850fd-be72-4b83-8036-30b31f441cb2 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:49.692028 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98f850fd-be72-4b83-8036-30b31f441cb2 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:49.715601 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 625/1259] 10.222.0.52 () {56 vars in 1148 bytes} [Sun Sep 25 00:45:49 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:45:49.724989 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703467a5-4dcf-40c4-8009-91a0926bc122 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:49.770635 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:49.889042 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703467a5-4dcf-40c4-8009-91a0926bc122 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:49.957869 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-543c5ed4-7789-42c4-a4db-f970a5a45cb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:49.958771 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543c5ed4-7789-42c4-a4db-f970a5a45cb1 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:49.959098 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543c5ed4-7789-42c4-a4db-f970a5a45cb1 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:49.959584 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543c5ed4-7789-42c4-a4db-f970a5a45cb1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:49.961007 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-543c5ed4-7789-42c4-a4db-f970a5a45cb1 None admin] RBAC: Adding query filter params (role.id=2c242abf6ed8410ab0a4a354ca1794fb, scope.project.id=9db0fd07fc494203b2ada13f3cd046d3, user.id=11a80b55dcd444df901c8bf80f2a2c26) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:45:49.961747 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-543c5ed4-7789-42c4-a4db-f970a5a45cb1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:49.964464 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-543c5ed4-7789-42c4-a4db-f970a5a45cb1 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:49.975145 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 635/1260] 10.222.0.52 () {56 vars in 1374 bytes} [Sun Sep 25 00:45:49 2022] GET /identity/v3/role_assignments?user.id=11a80b55dcd444df901c8bf80f2a2c26&scope.project.id=9db0fd07fc494203b2ada13f3cd046d3&role.id=2c242abf6ed8410ab0a4a354ca1794fb => generated 259 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:45:52.347510 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f9d627-ef57-4158-827e-6ca374f4b040 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:52.348104 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f9d627-ef57-4158-827e-6ca374f4b040 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:52.348104 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f9d627-ef57-4158-827e-6ca374f4b040 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:52.349586 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 626/1261] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:45:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 00:45:52.359231 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfc2dbe-dd49-428d-a9a0-028c10c2347c None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:52.359802 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfc2dbe-dd49-428d-a9a0-028c10c2347c None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:52.360183 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfc2dbe-dd49-428d-a9a0-028c10c2347c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:52.389142 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-5bfc2dbe-dd49-428d-a9a0-028c10c2347c None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:45:52.390457 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-5bfc2dbe-dd49-428d-a9a0-028c10c2347c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:45:52.467884 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 636/1262] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:45:52 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:45:52.824479 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f9d627-ef57-4158-827e-6ca374f4b040 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:52.836383 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:52.932792 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f9d627-ef57-4158-827e-6ca374f4b040 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:52.974661 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8313299c-950f-43ce-91ef-e24a663c8e1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:52.975318 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8313299c-950f-43ce-91ef-e24a663c8e1e None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:52.975593 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8313299c-950f-43ce-91ef-e24a663c8e1e None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:52.975855 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8313299c-950f-43ce-91ef-e24a663c8e1e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:52.989107 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8313299c-950f-43ce-91ef-e24a663c8e1e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:45:52.994164 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-8313299c-950f-43ce-91ef-e24a663c8e1e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:45:52.994164 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:45:52.994164 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:45:52.994164 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:45:52.994164 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:45:52.994164 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:45:52.994164 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:45:52.994164 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:45:52.994164 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:45:52.994164 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:45:52.994164 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:45:52.994164 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:45:52.994164 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:45:52.994164 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:45:52.994164 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:45:52.994164 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:45:52.995810 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:45:52.995810 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:45:52.995810 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:45:52.995810 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:45:52.995810 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:45:52.995810 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:45:52.995810 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:45:52.995810 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:45:52.995810 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:45:52.995810 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:45:52.995810 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:45:52.995810 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:45:52.995810 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:45:52.995810 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:45:52.995810 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:45:52.995810 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:45:52.996740 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:45:52.996740 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:45:52.996740 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:45:52.996740 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:45:52.996740 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:45:52.996740 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:45:52.996740 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:45:52.996740 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:45:52.996740 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:45:52.996740 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:45:52.996740 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:45:52.996740 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:45:52.996740 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 627/1263] 10.222.0.52 () {56 vars in 1096 bytes} [Sun Sep 25 00:45:52 2022] GET /identity/v3/roles/admin => generated 83 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:45:53.002798 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bfc2dbe-dd49-428d-a9a0-028c10c2347c None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:53.023729 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:53.152844 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bfc2dbe-dd49-428d-a9a0-028c10c2347c None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:53.190338 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17318561-270e-46c5-92f0-6cb1b1790e96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:53.190948 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17318561-270e-46c5-92f0-6cb1b1790e96 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:53.191222 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17318561-270e-46c5-92f0-6cb1b1790e96 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:53.191490 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17318561-270e-46c5-92f0-6cb1b1790e96 None admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:53.192163 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17318561-270e-46c5-92f0-6cb1b1790e96 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:45:53.192462 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17318561-270e-46c5-92f0-6cb1b1790e96 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:53.193873 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17318561-270e-46c5-92f0-6cb1b1790e96 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:53.203661 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 637/1264] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:45:53 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:45:53.211817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8313299c-950f-43ce-91ef-e24a663c8e1e None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:53.284732 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:53.322052 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8313299c-950f-43ce-91ef-e24a663c8e1e None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:53.361600 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2ce458d-43d4-4937-bbc8-1dbb3dc8ebfa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:53.362326 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ce458d-43d4-4937-bbc8-1dbb3dc8ebfa None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:53.362505 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ce458d-43d4-4937-bbc8-1dbb3dc8ebfa None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:53.362771 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ce458d-43d4-4937-bbc8-1dbb3dc8ebfa None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:53.374511 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2ce458d-43d4-4937-bbc8-1dbb3dc8ebfa None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:53.376054 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2ce458d-43d4-4937-bbc8-1dbb3dc8ebfa None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:53.392466 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-d2ce458d-43d4-4937-bbc8-1dbb3dc8ebfa None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:45:53.392466 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:45:53.392466 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:45:53.392466 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:45:53.392466 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:45:53.392466 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:45:53.392466 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:45:53.392466 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:45:53.392466 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:45:53.392466 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:45:53.392466 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:45:53.392466 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:45:53.392466 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:45:53.392466 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:45:53.392466 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:45:53.392466 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:45:53.394358 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:45:53.394358 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:45:53.394358 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:45:53.394358 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:45:53.394358 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:45:53.394358 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:45:53.394358 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:45:53.394358 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:45:53.394358 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:45:53.394358 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:45:53.394358 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:45:53.394358 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:45:53.394358 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:45:53.394358 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:45:53.394358 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:45:53.394358 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:45:53.394358 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:45:53.396454 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:45:53.396454 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:45:53.396454 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:45:53.396454 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 628/1265] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:45:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:45:53.400863 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17318561-270e-46c5-92f0-6cb1b1790e96 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:53.535149 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:53.556276 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17318561-270e-46c5-92f0-6cb1b1790e96 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:53.594292 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c7ae60-95b2-4a40-812b-1b27c8c3f6ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:53.594908 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c7ae60-95b2-4a40-812b-1b27c8c3f6ee None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:53.595184 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c7ae60-95b2-4a40-812b-1b27c8c3f6ee None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:53.595451 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c7ae60-95b2-4a40-812b-1b27c8c3f6ee None admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:53.596086 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71c7ae60-95b2-4a40-812b-1b27c8c3f6ee None admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:45:53.596385 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71c7ae60-95b2-4a40-812b-1b27c8c3f6ee None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:53.597781 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71c7ae60-95b2-4a40-812b-1b27c8c3f6ee None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:53.611029 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 638/1266] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:45:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:45:53.619582 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2ce458d-43d4-4937-bbc8-1dbb3dc8ebfa None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:53.729716 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2ce458d-43d4-4937-bbc8-1dbb3dc8ebfa None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:53.752579 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:53.768851 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d5c38f-1069-4ec5-aae8-b5f672ac4f1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:53.769522 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d5c38f-1069-4ec5-aae8-b5f672ac4f1f None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:53.769801 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d5c38f-1069-4ec5-aae8-b5f672ac4f1f None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:53.770071 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d5c38f-1069-4ec5-aae8-b5f672ac4f1f None admin] PATH_INFO: `/v3/users/placement` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:53.779582 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d5c38f-1069-4ec5-aae8-b5f672ac4f1f None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:53.781033 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d5c38f-1069-4ec5-aae8-b5f672ac4f1f None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:53.793344 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-05d5c38f-1069-4ec5-aae8-b5f672ac4f1f None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 25 00:45:53.793344 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:45:53.793344 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:45:53.793344 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:45:53.793344 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:45:53.793344 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:45:53.793344 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:45:53.793344 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:45:53.793344 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:45:53.793344 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:45:53.793344 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:45:53.793344 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:45:53.793344 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 00:45:53.793344 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 00:45:53.793344 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 00:45:53.793344 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 00:45:53.796149 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:45:53.796149 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:45:53.796149 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 00:45:53.796149 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:45:53.796149 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 00:45:53.796149 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:45:53.796149 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:45:53.796149 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:45:53.796149 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:45:53.796149 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:45:53.796149 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:45:53.796149 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:45:53.796149 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:45:53.796149 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:45:53.796149 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:45:53.796149 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:45:53.796149 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:45:53.796149 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:45:53.798473 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:45:53.798473 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:45:53.798473 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 00:45:53.798473 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 00:45:53.798473 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 00:45:53.798473 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 00:45:53.798473 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 00:45:53.798473 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 00:45:53.798473 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 25 00:45:53.798473 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:45:53.798473 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 629/1267] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:45:53 2022] GET /identity/v3/users/placement => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:45:53.802173 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c7ae60-95b2-4a40-812b-1b27c8c3f6ee None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:53.980700 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c7ae60-95b2-4a40-812b-1b27c8c3f6ee None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:54.023059 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baff8e08-0dc8-4310-8588-e5d547f6d54e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:54.023570 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baff8e08-0dc8-4310-8588-e5d547f6d54e None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:54.023842 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baff8e08-0dc8-4310-8588-e5d547f6d54e None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:54.024103 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baff8e08-0dc8-4310-8588-e5d547f6d54e None admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:54.024825 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baff8e08-0dc8-4310-8588-e5d547f6d54e None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:45:54.025123 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baff8e08-0dc8-4310-8588-e5d547f6d54e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:54.026578 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baff8e08-0dc8-4310-8588-e5d547f6d54e None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:54.036722 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:54.060069 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 639/1268] 10.222.0.52 () {56 vars in 1143 bytes} [Sun Sep 25 00:45:53 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:45:54.061984 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d5c38f-1069-4ec5-aae8-b5f672ac4f1f None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:54.176538 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d5c38f-1069-4ec5-aae8-b5f672ac4f1f None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:54.212520 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49872c17-d22a-467e-ad99-d0077038fb22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:54.213140 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49872c17-d22a-467e-ad99-d0077038fb22 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:54.213440 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49872c17-d22a-467e-ad99-d0077038fb22 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:54.213704 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49872c17-d22a-467e-ad99-d0077038fb22 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:54.225366 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49872c17-d22a-467e-ad99-d0077038fb22 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:54.226800 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49872c17-d22a-467e-ad99-d0077038fb22 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:54.242293 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-49872c17-d22a-467e-ad99-d0077038fb22 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:45:54.242293 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:45:54.242293 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:45:54.242293 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:45:54.242293 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:45:54.242293 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:45:54.242293 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:45:54.242293 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:45:54.242293 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:45:54.242293 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:45:54.242293 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:45:54.242293 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:45:54.242293 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:45:54.242293 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:45:54.242293 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:45:54.242293 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:45:54.244122 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:45:54.244122 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:45:54.244122 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:45:54.244122 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:45:54.244122 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:45:54.244122 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:45:54.244122 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:45:54.244122 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:45:54.244122 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:45:54.244122 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:45:54.244122 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:45:54.244122 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:45:54.244122 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:45:54.244122 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:45:54.244122 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:45:54.244122 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:45:54.244122 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:45:54.245929 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:45:54.245929 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:45:54.245929 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:45:54.248882 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 630/1269] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:45:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:45:54.250179 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baff8e08-0dc8-4310-8588-e5d547f6d54e None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:54.358167 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baff8e08-0dc8-4310-8588-e5d547f6d54e None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:54.396502 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec65e696-73a6-4715-9599-f76a6a44c49b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:54.397158 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec65e696-73a6-4715-9599-f76a6a44c49b None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:54.397506 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec65e696-73a6-4715-9599-f76a6a44c49b None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:54.397825 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec65e696-73a6-4715-9599-f76a6a44c49b None admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:54.398506 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec65e696-73a6-4715-9599-f76a6a44c49b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:45:54.398863 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec65e696-73a6-4715-9599-f76a6a44c49b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:54.400293 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec65e696-73a6-4715-9599-f76a6a44c49b None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:54.413156 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 640/1270] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:45:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:45:54.419303 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49872c17-d22a-467e-ad99-d0077038fb22 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:54.454213 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:54.530771 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49872c17-d22a-467e-ad99-d0077038fb22 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:54.565872 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b7922dc-8ddc-41a8-9be8-bb3dc66bd816 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:54.566517 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7922dc-8ddc-41a8-9be8-bb3dc66bd816 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:54.566796 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7922dc-8ddc-41a8-9be8-bb3dc66bd816 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:54.567067 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7922dc-8ddc-41a8-9be8-bb3dc66bd816 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:54.574089 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b7922dc-8ddc-41a8-9be8-bb3dc66bd816 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:54.576146 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b7922dc-8ddc-41a8-9be8-bb3dc66bd816 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:54.587973 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-6b7922dc-8ddc-41a8-9be8-bb3dc66bd816 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:45:54.587973 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:45:54.587973 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:45:54.587973 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:45:54.587973 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:45:54.587973 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:45:54.587973 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:45:54.587973 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:45:54.587973 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:45:54.587973 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:45:54.587973 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:45:54.587973 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:45:54.587973 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 00:45:54.587973 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 00:45:54.587973 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 00:45:54.589249 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 00:45:54.589249 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:45:54.589249 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:45:54.589249 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:45:54.589249 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:45:54.589249 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:45:54.589249 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:45:54.589249 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:45:54.589249 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:45:54.589249 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:45:54.589249 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:45:54.589249 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:45:54.589249 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:45:54.589249 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:45:54.589249 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:45:54.589249 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:45:54.589249 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:45:54.591518 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 00:45:54.591518 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 00:45:54.591518 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 00:45:54.591518 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 00:45:54.591518 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 00:45:54.591518 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 00:45:54.591518 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:45:54.591518 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:45:54.591518 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 631/1271] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:45:54 2022] GET /identity/v3/projects/service => generated 88 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:45:54.596264 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec65e696-73a6-4715-9599-f76a6a44c49b None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:54.659168 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:54.703654 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec65e696-73a6-4715-9599-f76a6a44c49b None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:54.739549 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eaba398-388a-4aa9-9ba1-e303e69e9c14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:54.740151 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eaba398-388a-4aa9-9ba1-e303e69e9c14 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:54.740424 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eaba398-388a-4aa9-9ba1-e303e69e9c14 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:54.740687 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eaba398-388a-4aa9-9ba1-e303e69e9c14 None admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:54.741407 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eaba398-388a-4aa9-9ba1-e303e69e9c14 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:45:54.741708 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eaba398-388a-4aa9-9ba1-e303e69e9c14 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:54.743122 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eaba398-388a-4aa9-9ba1-e303e69e9c14 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:54.759368 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 641/1272] 10.222.0.52 () {56 vars in 1148 bytes} [Sun Sep 25 00:45:54 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:45:54.766107 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b7922dc-8ddc-41a8-9be8-bb3dc66bd816 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:54.864833 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:54.879854 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b7922dc-8ddc-41a8-9be8-bb3dc66bd816 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:54.918476 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7ca53b-78eb-4205-bbfe-7de21269e0a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:54.919095 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7ca53b-78eb-4205-bbfe-7de21269e0a0 None admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:54.919372 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7ca53b-78eb-4205-bbfe-7de21269e0a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:54.919642 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7ca53b-78eb-4205-bbfe-7de21269e0a0 None admin] PATH_INFO: `/v3/projects/9db0fd07fc494203b2ada13f3cd046d3/users/11a80b55dcd444df901c8bf80f2a2c26/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:54.921614 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e7ca53b-78eb-4205-bbfe-7de21269e0a0 None admin] RBAC: Authorizing `identity:create_grant(project_id=9db0fd07fc494203b2ada13f3cd046d3, user_id=11a80b55dcd444df901c8bf80f2a2c26, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:54.923061 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e7ca53b-78eb-4205-bbfe-7de21269e0a0 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:54.937776 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 632/1273] 10.222.0.52 () {58 vars in 1439 bytes} [Sun Sep 25 00:45:54 2022] PUT /identity/v3/projects/9db0fd07fc494203b2ada13f3cd046d3/users/11a80b55dcd444df901c8bf80f2a2c26/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:45:57.192360 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03a91ac-b774-4bbf-8915-e6b5396a3c27 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:57.193215 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03a91ac-b774-4bbf-8915-e6b5396a3c27 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:57.193215 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03a91ac-b774-4bbf-8915-e6b5396a3c27 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:57.194558 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 642/1274] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:45:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 00:45:57.200184 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe45806-da39-43da-99ed-b32337d7ad26 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:57.200883 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe45806-da39-43da-99ed-b32337d7ad26 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:57.200883 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe45806-da39-43da-99ed-b32337d7ad26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:57.220845 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-ebe45806-da39-43da-99ed-b32337d7ad26 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:45:57.222099 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-ebe45806-da39-43da-99ed-b32337d7ad26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:45:57.317320 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 633/1275] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:45:57 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 25 00:45:57.618017 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e03a91ac-b774-4bbf-8915-e6b5396a3c27 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:57.730530 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e03a91ac-b774-4bbf-8915-e6b5396a3c27 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:57.758324 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:57.777141 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d17eee5-a9b3-4549-afce-d2785f847b80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:57.777815 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d17eee5-a9b3-4549-afce-d2785f847b80 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:57.778357 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d17eee5-a9b3-4549-afce-d2785f847b80 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:57.778737 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d17eee5-a9b3-4549-afce-d2785f847b80 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:57.789557 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d17eee5-a9b3-4549-afce-d2785f847b80 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 00:45:57.794940 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-1d17eee5-a9b3-4549-afce-d2785f847b80 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:45:57.794940 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:45:57.794940 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:45:57.794940 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:45:57.794940 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:45:57.794940 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:45:57.794940 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:45:57.794940 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:45:57.794940 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:45:57.794940 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:45:57.794940 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:45:57.794940 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:45:57.794940 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 00:45:57.794940 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 00:45:57.794940 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 00:45:57.794940 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 00:45:57.796724 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 00:45:57.796724 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 00:45:57.796724 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 00:45:57.796724 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 00:45:57.796724 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:45:57.796724 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:45:57.796724 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:45:57.796724 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:45:57.796724 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:45:57.796724 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:45:57.796724 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:45:57.796724 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:45:57.796724 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:45:57.796724 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:45:57.796724 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:45:57.796724 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:45:57.798640 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:45:57.798640 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:45:57.798640 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:45:57.798640 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:45:57.798640 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 00:45:57.798640 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 00:45:57.798640 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 00:45:57.798640 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 00:45:57.798640 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 00:45:57.798640 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 00:45:57.798640 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 00:45:57.798640 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:45:57.798640 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 643/1276] 10.222.0.52 () {56 vars in 1096 bytes} [Sun Sep 25 00:45:57 2022] GET /identity/v3/roles/admin => generated 83 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:45:57.804626 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe45806-da39-43da-99ed-b32337d7ad26 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:57.907337 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:57.928194 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe45806-da39-43da-99ed-b32337d7ad26 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:57.965319 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb1e0c72-e85c-44a3-b8fc-411424b3c83f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:57.966448 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb1e0c72-e85c-44a3-b8fc-411424b3c83f None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:57.966721 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb1e0c72-e85c-44a3-b8fc-411424b3c83f None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:57.966990 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb1e0c72-e85c-44a3-b8fc-411424b3c83f None admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:57.967651 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb1e0c72-e85c-44a3-b8fc-411424b3c83f None admin] RBAC: Adding query filter params (name=admin) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:45:57.967949 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb1e0c72-e85c-44a3-b8fc-411424b3c83f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:57.969344 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb1e0c72-e85c-44a3-b8fc-411424b3c83f None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:57.977685 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 634/1277] 10.222.0.52 () {56 vars in 1099 bytes} [Sun Sep 25 00:45:57 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:45:57.984874 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d17eee5-a9b3-4549-afce-d2785f847b80 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:58.141633 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d17eee5-a9b3-4549-afce-d2785f847b80 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:58.178989 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6980a99a-929d-4cb9-a00b-e5430bfb8175 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:58.179623 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6980a99a-929d-4cb9-a00b-e5430bfb8175 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:58.179896 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6980a99a-929d-4cb9-a00b-e5430bfb8175 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:58.180161 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6980a99a-929d-4cb9-a00b-e5430bfb8175 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:58.192243 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6980a99a-929d-4cb9-a00b-e5430bfb8175 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:58.193759 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6980a99a-929d-4cb9-a00b-e5430bfb8175 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:58.209448 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-6980a99a-929d-4cb9-a00b-e5430bfb8175 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:45:58.209448 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:45:58.209448 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:45:58.209448 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:45:58.209448 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:45:58.209448 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:45:58.209448 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:45:58.209448 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:45:58.209448 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:45:58.209448 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:45:58.209448 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:45:58.209448 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:45:58.209448 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:45:58.209448 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:45:58.209448 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:45:58.209448 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:45:58.211381 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:45:58.211381 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:45:58.211381 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:45:58.211381 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:45:58.211381 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:45:58.211381 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:45:58.211381 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:45:58.211381 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:45:58.211381 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:45:58.211381 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:45:58.211381 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:45:58.211381 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:45:58.211381 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:45:58.211381 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:45:58.211381 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:45:58.211381 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:45:58.211381 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:45:58.213142 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:45:58.213142 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:45:58.213142 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:45:58.213142 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 644/1278] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:45:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:45:58.218293 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb1e0c72-e85c-44a3-b8fc-411424b3c83f None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:58.334499 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb1e0c72-e85c-44a3-b8fc-411424b3c83f None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:58.373790 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eae5615c-4c22-44cc-8502-a1253990fca4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:58.374419 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae5615c-4c22-44cc-8502-a1253990fca4 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:58.374695 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae5615c-4c22-44cc-8502-a1253990fca4 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:58.374962 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae5615c-4c22-44cc-8502-a1253990fca4 None admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:58.375621 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eae5615c-4c22-44cc-8502-a1253990fca4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:45:58.375919 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eae5615c-4c22-44cc-8502-a1253990fca4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:58.377353 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eae5615c-4c22-44cc-8502-a1253990fca4 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:58.392184 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 635/1279] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:45:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:45:58.398252 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6980a99a-929d-4cb9-a00b-e5430bfb8175 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:58.433559 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:58.515048 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6980a99a-929d-4cb9-a00b-e5430bfb8175 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:58.552939 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a80fd5-4ea1-4f9d-a409-b3d9335d3a7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:58.553617 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a80fd5-4ea1-4f9d-a409-b3d9335d3a7d None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:58.553893 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a80fd5-4ea1-4f9d-a409-b3d9335d3a7d None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:58.554161 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a80fd5-4ea1-4f9d-a409-b3d9335d3a7d None admin] PATH_INFO: `/v3/users/placement` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:58.563087 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37a80fd5-4ea1-4f9d-a409-b3d9335d3a7d None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:58.564541 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37a80fd5-4ea1-4f9d-a409-b3d9335d3a7d None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:58.577631 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-37a80fd5-4ea1-4f9d-a409-b3d9335d3a7d None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 25 00:45:58.577631 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:45:58.577631 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:45:58.577631 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:45:58.577631 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:45:58.577631 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:45:58.577631 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:45:58.577631 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:45:58.577631 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:45:58.577631 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:45:58.577631 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:45:58.577631 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:45:58.577631 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 00:45:58.577631 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 00:45:58.577631 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 00:45:58.577631 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 00:45:58.579281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:45:58.579281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:45:58.579281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 00:45:58.579281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:45:58.579281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 00:45:58.579281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 00:45:58.579281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:45:58.579281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:45:58.579281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:45:58.579281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:45:58.579281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:45:58.579281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:45:58.579281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:45:58.579281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:45:58.579281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:45:58.579281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:45:58.579281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:45:58.579281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:45:58.580747 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:45:58.580747 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:45:58.580747 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 00:45:58.580747 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 00:45:58.580747 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 00:45:58.580747 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 00:45:58.580747 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 00:45:58.580747 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 00:45:58.580747 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 25 00:45:58.580747 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:45:58.580747 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 645/1280] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:45:58 2022] GET /identity/v3/users/placement => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:45:58.584920 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eae5615c-4c22-44cc-8502-a1253990fca4 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:58.588983 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:58.701579 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eae5615c-4c22-44cc-8502-a1253990fca4 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:58.738964 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a6951d7-fe85-4f88-b3e6-481754dc6959 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:58.739573 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6951d7-fe85-4f88-b3e6-481754dc6959 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:58.739852 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6951d7-fe85-4f88-b3e6-481754dc6959 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:58.740123 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6951d7-fe85-4f88-b3e6-481754dc6959 None admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:58.740858 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a6951d7-fe85-4f88-b3e6-481754dc6959 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:45:58.741163 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a6951d7-fe85-4f88-b3e6-481754dc6959 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:58.742636 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a6951d7-fe85-4f88-b3e6-481754dc6959 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:58.762579 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 636/1281] 10.222.0.52 () {56 vars in 1143 bytes} [Sun Sep 25 00:45:58 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:45:58.768643 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a80fd5-4ea1-4f9d-a409-b3d9335d3a7d None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:58.869510 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:58.880292 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a80fd5-4ea1-4f9d-a409-b3d9335d3a7d None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:58.917329 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a49aab-2a85-4455-999c-b00191442e10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:58.917940 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a49aab-2a85-4455-999c-b00191442e10 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:58.918215 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a49aab-2a85-4455-999c-b00191442e10 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:58.918537 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a49aab-2a85-4455-999c-b00191442e10 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:58.992966 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43a49aab-2a85-4455-999c-b00191442e10 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:58.994444 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43a49aab-2a85-4455-999c-b00191442e10 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:59.094778 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:59.105776 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 646/1282] 10.222.0.52 () {58 vars in 1282 bytes} [Sun Sep 25 00:45:58 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 25 00:45:59.115728 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a6951d7-fe85-4f88-b3e6-481754dc6959 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:59.181884 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:59.226430 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a6951d7-fe85-4f88-b3e6-481754dc6959 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:59.263970 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133629d1-3abf-42c6-a45e-a41c38a3d2ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:59.264606 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133629d1-3abf-42c6-a45e-a41c38a3d2ed None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:59.264949 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133629d1-3abf-42c6-a45e-a41c38a3d2ed None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:59.265283 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133629d1-3abf-42c6-a45e-a41c38a3d2ed None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:59.276654 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-133629d1-3abf-42c6-a45e-a41c38a3d2ed None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:59.278734 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-133629d1-3abf-42c6-a45e-a41c38a3d2ed None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:59.295723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-133629d1-3abf-42c6-a45e-a41c38a3d2ed None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:45:59.295723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:45:59.295723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:45:59.295723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:45:59.295723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:45:59.295723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:45:59.295723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:45:59.295723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:45:59.295723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:45:59.295723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:45:59.295723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:45:59.295723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:45:59.295723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 00:45:59.295723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 00:45:59.295723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 00:45:59.295723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 00:45:59.301507 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:45:59.301507 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:45:59.301507 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:45:59.301507 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:45:59.301507 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:45:59.301507 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:45:59.301507 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:45:59.301507 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:45:59.301507 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:45:59.301507 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:45:59.301507 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:45:59.301507 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:45:59.301507 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:45:59.301507 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:45:59.301507 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:45:59.301507 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:45:59.301507 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 00:45:59.303189 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 00:45:59.303189 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 00:45:59.303189 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:45:59.303189 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 637/1283] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:45:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:45:59.303575 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a49aab-2a85-4455-999c-b00191442e10 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:59.409422 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a49aab-2a85-4455-999c-b00191442e10 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:59.447442 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a81fda-df2a-4bce-ab6e-06250c027ba9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:59.448453 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a81fda-df2a-4bce-ab6e-06250c027ba9 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:59.448744 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a81fda-df2a-4bce-ab6e-06250c027ba9 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:59.449036 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a81fda-df2a-4bce-ab6e-06250c027ba9 None admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:59.449916 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a81fda-df2a-4bce-ab6e-06250c027ba9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:45:59.450420 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a81fda-df2a-4bce-ab6e-06250c027ba9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:59.453279 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a81fda-df2a-4bce-ab6e-06250c027ba9 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:59.468969 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 647/1284] 10.222.0.52 () {56 vars in 1109 bytes} [Sun Sep 25 00:45:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:45:59.475266 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133629d1-3abf-42c6-a45e-a41c38a3d2ed None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:59.599276 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133629d1-3abf-42c6-a45e-a41c38a3d2ed None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:59.617280 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:59.643398 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36acacf4-ea67-4352-8033-3bb63bed24f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:59.644171 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36acacf4-ea67-4352-8033-3bb63bed24f6 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:59.644530 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36acacf4-ea67-4352-8033-3bb63bed24f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:59.644912 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36acacf4-ea67-4352-8033-3bb63bed24f6 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:59.652805 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36acacf4-ea67-4352-8033-3bb63bed24f6 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:59.654951 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36acacf4-ea67-4352-8033-3bb63bed24f6 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:59.668172 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application [None req-36acacf4-ea67-4352-8033-3bb63bed24f6 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:45:59.668172 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:45:59.668172 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:45:59.668172 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:45:59.668172 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:45:59.668172 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:45:59.668172 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:45:59.668172 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:45:59.668172 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:45:59.668172 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:45:59.668172 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:45:59.668172 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:45:59.668172 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 00:45:59.668172 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 00:45:59.668172 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 00:45:59.669341 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 00:45:59.669341 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:45:59.669341 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:45:59.669341 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:45:59.669341 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:45:59.669341 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:45:59.669341 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:45:59.669341 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:45:59.669341 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application with Lock( Sep 25 00:45:59.669341 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:45:59.669341 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:45:59.669341 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:45:59.669341 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:45:59.669341 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:45:59.669341 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:45:59.669341 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:45:59.669341 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:45:59.670554 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 00:45:59.670554 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 00:45:59.670554 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 00:45:59.670554 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 00:45:59.670554 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 00:45:59.670554 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 00:45:59.670554 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 00:45:59.670554 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: ERROR keystone.server.flask.application Sep 25 00:45:59.670554 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 638/1285] 10.222.0.52 () {56 vars in 1111 bytes} [Sun Sep 25 00:45:59 2022] GET /identity/v3/projects/service => generated 88 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:45:59.675870 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a81fda-df2a-4bce-ab6e-06250c027ba9 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:59.732590 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:45:59.789349 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a81fda-df2a-4bce-ab6e-06250c027ba9 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:45:59.826866 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f4bb16-cb52-4aa6-ba16-56541edc0d8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:45:59.827524 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f4bb16-cb52-4aa6-ba16-56541edc0d8e None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:45:59.827834 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f4bb16-cb52-4aa6-ba16-56541edc0d8e None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:45:59.828137 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f4bb16-cb52-4aa6-ba16-56541edc0d8e None admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:45:59.828862 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8f4bb16-cb52-4aa6-ba16-56541edc0d8e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:45:59.829199 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8f4bb16-cb52-4aa6-ba16-56541edc0d8e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:45:59.830686 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8f4bb16-cb52-4aa6-ba16-56541edc0d8e None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:45:59.844192 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 648/1286] 10.222.0.52 () {56 vars in 1148 bytes} [Sun Sep 25 00:45:59 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:45:59.854093 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36acacf4-ea67-4352-8033-3bb63bed24f6 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:45:59.973330 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36acacf4-ea67-4352-8033-3bb63bed24f6 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:46:00.014192 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca887698-dbff-4d04-842c-31e02f53a819 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:46:00.014865 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca887698-dbff-4d04-842c-31e02f53a819 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:46:00.015181 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca887698-dbff-4d04-842c-31e02f53a819 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:46:00.015491 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca887698-dbff-4d04-842c-31e02f53a819 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:46:00.016283 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca887698-dbff-4d04-842c-31e02f53a819 None admin] RBAC: Adding query filter params (role.id=2c242abf6ed8410ab0a4a354ca1794fb, scope.project.id=9db0fd07fc494203b2ada13f3cd046d3, user.id=11a80b55dcd444df901c8bf80f2a2c26) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:46:00.016629 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca887698-dbff-4d04-842c-31e02f53a819 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:46:00.018181 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca887698-dbff-4d04-842c-31e02f53a819 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:46:00.025222 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 639/1287] 10.222.0.52 () {56 vars in 1374 bytes} [Sun Sep 25 00:45:59 2022] GET /identity/v3/role_assignments?user.id=11a80b55dcd444df901c8bf80f2a2c26&scope.project.id=9db0fd07fc494203b2ada13f3cd046d3&role.id=2c242abf6ed8410ab0a4a354ca1794fb => generated 608 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:46:02.327508 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317b7654-28b7-453c-8d5e-ff2796e7285f None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:46:02.328084 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317b7654-28b7-453c-8d5e-ff2796e7285f None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:46:02.328084 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317b7654-28b7-453c-8d5e-ff2796e7285f None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:46:02.329373 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 649/1288] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:46:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:46:02.334928 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f5e015-6998-4364-8d49-84bd16ce02a9 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:46:02.335404 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f5e015-6998-4364-8d49-84bd16ce02a9 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:46:02.335561 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f5e015-6998-4364-8d49-84bd16ce02a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:46:02.354788 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-24f5e015-6998-4364-8d49-84bd16ce02a9 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:46:02.356662 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-24f5e015-6998-4364-8d49-84bd16ce02a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:46:02.391006 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:46:02.433676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 640/1289] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:46:02 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:46:02.704064 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-317b7654-28b7-453c-8d5e-ff2796e7285f None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:46:02.815726 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-317b7654-28b7-453c-8d5e-ff2796e7285f None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:46:02.829273 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:46:02.858039 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1317b1fc-a75c-4ef0-9c3e-683dd422e37c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:46:02.858710 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1317b1fc-a75c-4ef0-9c3e-683dd422e37c None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:46:02.859028 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1317b1fc-a75c-4ef0-9c3e-683dd422e37c None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:46:02.859379 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1317b1fc-a75c-4ef0-9c3e-683dd422e37c None admin] PATH_INFO: `/v3/services/placement` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:46:02.860011 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1317b1fc-a75c-4ef0-9c3e-683dd422e37c None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:46:02.861731 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1317b1fc-a75c-4ef0-9c3e-683dd422e37c None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:46:02.871671 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-1317b1fc-a75c-4ef0-9c3e-683dd422e37c None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 25 00:46:02.871671 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:46:02.871671 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 00:46:02.871671 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 00:46:02.871671 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 00:46:02.871671 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 00:46:02.871671 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 00:46:02.871671 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:46:02.871671 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 25 00:46:02.871671 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:46:02.871671 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 00:46:02.871671 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:46:02.871671 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:46:02.871671 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:46:02.871671 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:46:02.871671 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:46:02.871671 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:46:02.873653 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:46:02.873653 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:46:02.873653 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:46:02.873653 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:46:02.873653 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:46:02.873653 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:46:02.873653 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 00:46:02.873653 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 00:46:02.873653 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 00:46:02.873653 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 00:46:02.873653 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:46:02.873653 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:46:02.873653 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:46:02.873653 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:46:02.873653 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:46:02.873653 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:46:02.875262 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:46:02.875262 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:46:02.875262 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:46:02.875262 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:46:02.875262 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:46:02.875262 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:46:02.875262 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:46:02.875262 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:46:02.875262 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:46:02.875262 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:46:02.875262 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 00:46:02.875262 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:46:02.875262 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 25 00:46:02.875262 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:46:02.877503 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 650/1290] 10.222.0.52 () {56 vars in 1117 bytes} [Sun Sep 25 00:46:02 2022] GET /identity/v3/services/placement => generated 90 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 00:46:02.879178 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f5e015-6998-4364-8d49-84bd16ce02a9 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:46:02.999600 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f5e015-6998-4364-8d49-84bd16ce02a9 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:46:03.041495 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb12f650-daca-4347-9e7a-bc9023ce0734 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:46:03.042234 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb12f650-daca-4347-9e7a-bc9023ce0734 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:46:03.042508 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb12f650-daca-4347-9e7a-bc9023ce0734 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:46:03.042862 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb12f650-daca-4347-9e7a-bc9023ce0734 None admin] PATH_INFO: `/v3/services` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:46:03.044792 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb12f650-daca-4347-9e7a-bc9023ce0734 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:46:03.045177 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb12f650-daca-4347-9e7a-bc9023ce0734 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:46:03.046789 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb12f650-daca-4347-9e7a-bc9023ce0734 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:46:03.057422 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 641/1291] 10.222.0.52 () {56 vars in 1116 bytes} [Sun Sep 25 00:46:02 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:46:03.060540 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1317b1fc-a75c-4ef0-9c3e-683dd422e37c None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:46:03.183524 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1317b1fc-a75c-4ef0-9c3e-683dd422e37c None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:46:03.230078 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-510ced5c-f0fa-4221-8f65-162372b84b9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:46:03.231069 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510ced5c-f0fa-4221-8f65-162372b84b9e None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:46:03.231503 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510ced5c-f0fa-4221-8f65-162372b84b9e None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:46:03.231931 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510ced5c-f0fa-4221-8f65-162372b84b9e None admin] PATH_INFO: `/v3/services` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:46:03.232947 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-510ced5c-f0fa-4221-8f65-162372b84b9e None admin] RBAC: Adding query filter params (type=placement) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:46:03.233466 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-510ced5c-f0fa-4221-8f65-162372b84b9e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:46:03.236011 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-510ced5c-f0fa-4221-8f65-162372b84b9e None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:46:03.243656 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 651/1292] 10.222.0.52 () {56 vars in 1116 bytes} [Sun Sep 25 00:46:03 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:46:05.487273 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c86428-dbc1-417d-b059-c1e008c103e2 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:46:05.487918 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c86428-dbc1-417d-b059-c1e008c103e2 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:46:05.487918 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c86428-dbc1-417d-b059-c1e008c103e2 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:46:05.489083 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 642/1293] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:46:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:46:05.495199 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-162392fb-60e1-4d1d-a152-f4faa28df1f1 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:46:05.496034 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-162392fb-60e1-4d1d-a152-f4faa28df1f1 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:46:05.496034 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-162392fb-60e1-4d1d-a152-f4faa28df1f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:46:05.516067 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-162392fb-60e1-4d1d-a152-f4faa28df1f1 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:46:05.517343 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-162392fb-60e1-4d1d-a152-f4faa28df1f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:46:05.553538 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:46:05.594322 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 652/1294] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:46:05 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:46:05.875030 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c86428-dbc1-417d-b059-c1e008c103e2 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:46:05.908629 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:46:05.987516 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c86428-dbc1-417d-b059-c1e008c103e2 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:46:06.024670 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3bb031a-fd00-48fc-a513-05e91b37a571 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:46:06.025315 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bb031a-fd00-48fc-a513-05e91b37a571 None admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:46:06.025590 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bb031a-fd00-48fc-a513-05e91b37a571 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:46:06.025859 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bb031a-fd00-48fc-a513-05e91b37a571 None admin] PATH_INFO: `/v3/services` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:46:06.026513 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3bb031a-fd00-48fc-a513-05e91b37a571 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:46:06.028452 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3bb031a-fd00-48fc-a513-05e91b37a571 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:46:06.045355 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 643/1295] 10.222.0.52 () {60 vars in 1141 bytes} [Sun Sep 25 00:46:05 2022] POST /identity/v3/services => generated 246 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:46:08.462510 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac609d1-e895-4935-ba74-ff9903d68fee None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:46:08.463090 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac609d1-e895-4935-ba74-ff9903d68fee None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:46:08.463090 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac609d1-e895-4935-ba74-ff9903d68fee None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:46:08.464593 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 653/1296] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:46:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:46:08.471195 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56274620-1054-45e7-9999-e2d0a79ccfab None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:46:08.471991 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56274620-1054-45e7-9999-e2d0a79ccfab None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:46:08.471991 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56274620-1054-45e7-9999-e2d0a79ccfab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:46:08.490789 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-56274620-1054-45e7-9999-e2d0a79ccfab None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:46:08.491988 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-56274620-1054-45e7-9999-e2d0a79ccfab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:46:08.573132 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 644/1297] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:46:08 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 00:46:08.850705 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac609d1-e895-4935-ba74-ff9903d68fee None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:46:08.965692 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac609d1-e895-4935-ba74-ff9903d68fee None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:46:09.002513 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30902055-83f2-44e2-b0ff-082c2f255865 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:46:09.003144 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30902055-83f2-44e2-b0ff-082c2f255865 None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:46:09.003418 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30902055-83f2-44e2-b0ff-082c2f255865 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:46:09.003681 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30902055-83f2-44e2-b0ff-082c2f255865 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:46:09.004264 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30902055-83f2-44e2-b0ff-082c2f255865 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:46:09.005663 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30902055-83f2-44e2-b0ff-082c2f255865 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:46:09.015474 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-30902055-83f2-44e2-b0ff-082c2f255865 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 25 00:46:09.015474 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:46:09.015474 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 00:46:09.015474 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 00:46:09.015474 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 00:46:09.015474 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 00:46:09.015474 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 00:46:09.015474 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:46:09.015474 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 25 00:46:09.015474 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:46:09.015474 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 00:46:09.015474 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:46:09.015474 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:46:09.015474 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:46:09.015474 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:46:09.015474 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:46:09.015474 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:46:09.017415 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:46:09.017415 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:46:09.017415 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:46:09.017415 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:46:09.017415 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:46:09.017415 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:46:09.017415 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 00:46:09.017415 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 00:46:09.017415 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 00:46:09.017415 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 00:46:09.017415 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:46:09.017415 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:46:09.017415 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:46:09.017415 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:46:09.017415 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:46:09.017415 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:46:09.018972 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:46:09.018972 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:46:09.018972 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:46:09.018972 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:46:09.018972 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:46:09.018972 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:46:09.018972 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:46:09.018972 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:46:09.018972 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:46:09.018972 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:46:09.018972 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 00:46:09.018972 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:46:09.018972 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 25 00:46:09.018972 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:46:09.018972 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 654/1298] 10.222.0.52 () {56 vars in 1117 bytes} [Sun Sep 25 00:46:08 2022] GET /identity/v3/services/placement => generated 90 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:46:09.023449 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56274620-1054-45e7-9999-e2d0a79ccfab None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:46:09.057139 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:46:09.202606 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56274620-1054-45e7-9999-e2d0a79ccfab None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:46:09.265324 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1eba9e4-424a-457f-a875-e605c7f483e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:46:09.266331 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1eba9e4-424a-457f-a875-e605c7f483e9 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:46:09.266807 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1eba9e4-424a-457f-a875-e605c7f483e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:46:09.267268 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1eba9e4-424a-457f-a875-e605c7f483e9 None admin] PATH_INFO: `/v3/services` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:46:09.268385 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1eba9e4-424a-457f-a875-e605c7f483e9 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:46:09.268904 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1eba9e4-424a-457f-a875-e605c7f483e9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:46:09.271409 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1eba9e4-424a-457f-a875-e605c7f483e9 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:46:09.284114 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 645/1299] 10.222.0.52 () {56 vars in 1116 bytes} [Sun Sep 25 00:46:09 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:46:09.290318 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30902055-83f2-44e2-b0ff-082c2f255865 None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:46:09.327435 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:46:09.411495 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30902055-83f2-44e2-b0ff-082c2f255865 None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:46:09.449505 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed3d5de-c945-4955-ae71-8de7e53c299c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:46:09.450117 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed3d5de-c945-4955-ae71-8de7e53c299c None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:46:09.450387 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed3d5de-c945-4955-ae71-8de7e53c299c None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:46:09.450649 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed3d5de-c945-4955-ae71-8de7e53c299c None admin] PATH_INFO: `/v3/endpoints` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:46:09.451352 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed3d5de-c945-4955-ae71-8de7e53c299c None admin] RBAC: Adding query filter params (interface=public, service_id=c97eceaf329f4067a4c84c3700051ccf, region_id=RegionOne) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:46:09.451653 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed3d5de-c945-4955-ae71-8de7e53c299c None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:46:09.453082 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed3d5de-c945-4955-ae71-8de7e53c299c None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:46:09.459208 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 655/1300] 10.222.0.52 () {56 vars in 1251 bytes} [Sun Sep 25 00:46:09 2022] GET /identity/v3/endpoints?service_id=c97eceaf329f4067a4c84c3700051ccf&interface=public®ion_id=RegionOne => generated 194 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:46:09.469242 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1eba9e4-424a-457f-a875-e605c7f483e9 None admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:46:09.585551 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:46:09.612195 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1eba9e4-424a-457f-a875-e605c7f483e9 None admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:46:09.650152 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0003f3db-41ab-4f40-914d-fa73ecd5f1b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:46:09.650761 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0003f3db-41ab-4f40-914d-fa73ecd5f1b8 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:46:09.651038 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0003f3db-41ab-4f40-914d-fa73ecd5f1b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:46:09.651306 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0003f3db-41ab-4f40-914d-fa73ecd5f1b8 None admin] PATH_INFO: `/v3/services` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:46:09.651911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0003f3db-41ab-4f40-914d-fa73ecd5f1b8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:46:09.653637 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0003f3db-41ab-4f40-914d-fa73ecd5f1b8 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:46:09.660926 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 646/1301] 10.222.0.52 () {56 vars in 1087 bytes} [Sun Sep 25 00:46:09 2022] GET /identity/v3/services => generated 2191 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:46:11.877662 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5427c5fd-bf09-41cc-8771-fd5d0be6c4e9 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:46:11.878444 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5427c5fd-bf09-41cc-8771-fd5d0be6c4e9 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:46:11.878444 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5427c5fd-bf09-41cc-8771-fd5d0be6c4e9 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:46:11.879598 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 656/1302] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:46:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:46:11.885457 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d65b7b-62da-46a1-9c5f-84ee24a7de84 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:46:11.886218 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d65b7b-62da-46a1-9c5f-84ee24a7de84 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:46:11.886218 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d65b7b-62da-46a1-9c5f-84ee24a7de84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:46:11.905182 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-04d65b7b-62da-46a1-9c5f-84ee24a7de84 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:46:11.906412 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-04d65b7b-62da-46a1-9c5f-84ee24a7de84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:46:11.981217 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 647/1303] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:46:11 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 25 00:46:12.255161 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5427c5fd-bf09-41cc-8771-fd5d0be6c4e9 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:46:12.359278 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:46:12.367326 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5427c5fd-bf09-41cc-8771-fd5d0be6c4e9 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:46:12.402703 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad814d0-1cf5-4810-ad12-7b48ad26fe0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:46:12.403390 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad814d0-1cf5-4810-ad12-7b48ad26fe0b None admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:46:12.403715 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad814d0-1cf5-4810-ad12-7b48ad26fe0b None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:46:12.404017 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad814d0-1cf5-4810-ad12-7b48ad26fe0b None admin] PATH_INFO: `/v3/services/placement` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:46:12.404622 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fad814d0-1cf5-4810-ad12-7b48ad26fe0b None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:46:12.406085 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fad814d0-1cf5-4810-ad12-7b48ad26fe0b None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:46:12.415567 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application [None req-fad814d0-1cf5-4810-ad12-7b48ad26fe0b None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 25 00:46:12.415567 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:46:12.415567 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 00:46:12.415567 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 00:46:12.415567 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 00:46:12.415567 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 00:46:12.415567 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 00:46:12.415567 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:46:12.415567 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 25 00:46:12.415567 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:46:12.415567 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 00:46:12.415567 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:46:12.415567 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 00:46:12.415567 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 00:46:12.415567 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 00:46:12.415567 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 00:46:12.415567 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 00:46:12.416912 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 00:46:12.416912 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 00:46:12.416912 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 00:46:12.416912 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 00:46:12.416912 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 00:46:12.416912 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 00:46:12.416912 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 00:46:12.416912 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 00:46:12.416912 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 00:46:12.416912 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 00:46:12.416912 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 00:46:12.416912 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 00:46:12.416912 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 00:46:12.416912 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 00:46:12.416912 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 00:46:12.416912 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 00:46:12.418896 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 00:46:12.418896 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application with Lock( Sep 25 00:46:12.418896 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 00:46:12.418896 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self._enter() Sep 25 00:46:12.418896 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 00:46:12.418896 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 00:46:12.418896 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 00:46:12.418896 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application return self.creator() Sep 25 00:46:12.418896 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 00:46:12.418896 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application created_value = creator( Sep 25 00:46:12.418896 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 00:46:12.418896 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 00:46:12.418896 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 25 00:46:12.418896 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: ERROR keystone.server.flask.application Sep 25 00:46:12.418896 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 657/1304] 10.222.0.52 () {56 vars in 1117 bytes} [Sun Sep 25 00:46:12 2022] GET /identity/v3/services/placement => generated 90 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 00:46:12.427395 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d65b7b-62da-46a1-9c5f-84ee24a7de84 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:46:12.514212 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:46:12.537866 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d65b7b-62da-46a1-9c5f-84ee24a7de84 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:46:12.572878 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b99c3c8-e3dc-427e-8ca9-4ab29f0fd731 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:46:12.573559 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b99c3c8-e3dc-427e-8ca9-4ab29f0fd731 None admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:46:12.573887 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b99c3c8-e3dc-427e-8ca9-4ab29f0fd731 None admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:46:12.574817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b99c3c8-e3dc-427e-8ca9-4ab29f0fd731 None admin] PATH_INFO: `/v3/services` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:46:12.575846 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b99c3c8-e3dc-427e-8ca9-4ab29f0fd731 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:46:12.576199 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b99c3c8-e3dc-427e-8ca9-4ab29f0fd731 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:46:12.577667 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b99c3c8-e3dc-427e-8ca9-4ab29f0fd731 None admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:46:12.582987 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 648/1305] 10.222.0.52 () {56 vars in 1116 bytes} [Sun Sep 25 00:46:12 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:46:12.588893 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad814d0-1cf5-4810-ad12-7b48ad26fe0b None admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:46:12.690398 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad814d0-1cf5-4810-ad12-7b48ad26fe0b None admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:46:12.724221 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd5a1bc-b138-4c8f-b298-10e86fdc7bc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:46:12.724810 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd5a1bc-b138-4c8f-b298-10e86fdc7bc2 None admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:46:12.725078 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd5a1bc-b138-4c8f-b298-10e86fdc7bc2 None admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:46:12.725365 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd5a1bc-b138-4c8f-b298-10e86fdc7bc2 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:46:12.726025 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddd5a1bc-b138-4c8f-b298-10e86fdc7bc2 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:46:12.727374 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddd5a1bc-b138-4c8f-b298-10e86fdc7bc2 None admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:46:12.748363 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 658/1306] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:46:12 2022] POST /identity/v3/endpoints => generated 331 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:46:22.580105 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=30 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:46:22.743793 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:47:20.878235 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed453b0-4d48-4f8f-8b31-1c82f51bfe69 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:47:20.878823 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed453b0-4d48-4f8f-8b31-1c82f51bfe69 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:47:20.878823 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed453b0-4d48-4f8f-8b31-1c82f51bfe69 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:47:20.880088 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 649/1307] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:47:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:47:20.886963 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c62a13-4b61-4dbf-8e35-0f871708e19c None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:47:20.887472 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c62a13-4b61-4dbf-8e35-0f871708e19c None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:47:20.887472 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c62a13-4b61-4dbf-8e35-0f871708e19c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:47:20.990942 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-b0c62a13-4b61-4dbf-8e35-0f871708e19c None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:47:20.993608 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-b0c62a13-4b61-4dbf-8e35-0f871708e19c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:47:21.129349 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 659/1308] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:47:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 240 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:47:21.432140 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dbcf67a-7ecd-4241-ac78-36c3924b4b81 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:47:21.432454 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dbcf67a-7ecd-4241-ac78-36c3924b4b81 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:47:21.432678 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dbcf67a-7ecd-4241-ac78-36c3924b4b81 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:47:21.434049 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 650/1309] 10.222.0.52 () {54 vars in 964 bytes} [Sun Sep 25 00:47:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:47:21.440226 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28aefb90-209e-47a1-9a6c-31c11b555c87 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:47:21.440949 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28aefb90-209e-47a1-9a6c-31c11b555c87 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:47:21.441213 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28aefb90-209e-47a1-9a6c-31c11b555c87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:47:21.534780 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-28aefb90-209e-47a1-9a6c-31c11b555c87 None None] MFA Rules not processed for user `20c274e96dec478b98ef51f80ea893b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:47:21.535872 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-28aefb90-209e-47a1-9a6c-31c11b555c87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:47:21.559360 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:47:21.622574 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 660/1310] 10.222.0.52 () {58 vars in 1061 bytes} [Sun Sep 25 00:47:21 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:47:21.631809 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dbcf67a-7ecd-4241-ac78-36c3924b4b81 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:47:21.633628 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-9dbcf67a-7ecd-4241-ac78-36c3924b4b81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:47:21.749168 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dbcf67a-7ecd-4241-ac78-36c3924b4b81 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:47:21.833386 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60d88057-09a2-4e6b-ac59-b4bc20adeeca service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:47:21.833742 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d88057-09a2-4e6b-ac59-b4bc20adeeca service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:47:21.834023 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d88057-09a2-4e6b-ac59-b4bc20adeeca service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:47:21.834253 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d88057-09a2-4e6b-ac59-b4bc20adeeca service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:47:21.835776 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-60d88057-09a2-4e6b-ac59-b4bc20adeeca service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:47:21.898774 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:47:21.963968 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60d88057-09a2-4e6b-ac59-b4bc20adeeca service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:47:21.965432 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60d88057-09a2-4e6b-ac59-b4bc20adeeca service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:47:21.987644 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 651/1311] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:47:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 356 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:47:24.512407 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-319fff61-04f9-4288-801d-41db944bc334 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:47:24.512993 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-319fff61-04f9-4288-801d-41db944bc334 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:47:24.512993 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-319fff61-04f9-4288-801d-41db944bc334 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:47:24.514346 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 661/1312] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:47:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:47:24.519723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2920d3-dd98-49f4-b2d7-3733bb116a50 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:47:24.520163 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2920d3-dd98-49f4-b2d7-3733bb116a50 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:47:24.520273 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2920d3-dd98-49f4-b2d7-3733bb116a50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:47:24.593767 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-1f2920d3-dd98-49f4-b2d7-3733bb116a50 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:47:24.594951 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-1f2920d3-dd98-49f4-b2d7-3733bb116a50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:47:24.696859 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 652/1313] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:47:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:47:24.994928 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-319fff61-04f9-4288-801d-41db944bc334 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:47:24.996515 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-319fff61-04f9-4288-801d-41db944bc334 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:47:25.115502 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-319fff61-04f9-4288-801d-41db944bc334 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:47:25.194361 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:47:25.198524 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee205fda-d892-45c9-9ea3-bc2bc96ca453 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:47:25.199155 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee205fda-d892-45c9-9ea3-bc2bc96ca453 service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:47:25.199624 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee205fda-d892-45c9-9ea3-bc2bc96ca453 service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:47:25.199920 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee205fda-d892-45c9-9ea3-bc2bc96ca453 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:47:25.201750 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-ee205fda-d892-45c9-9ea3-bc2bc96ca453 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:47:25.319819 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee205fda-d892-45c9-9ea3-bc2bc96ca453 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:47:25.321194 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee205fda-d892-45c9-9ea3-bc2bc96ca453 service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:47:25.339923 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 662/1314] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:47:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 346 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:47:27.853138 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918f88ff-6445-4518-b727-56c6395c3548 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:47:27.853717 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918f88ff-6445-4518-b727-56c6395c3548 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:47:27.853717 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918f88ff-6445-4518-b727-56c6395c3548 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:47:27.854970 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 653/1315] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:47:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:47:27.860595 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7105a0-46da-4b22-84d0-50684711c5fc None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:47:27.860992 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7105a0-46da-4b22-84d0-50684711c5fc None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:47:27.861139 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7105a0-46da-4b22-84d0-50684711c5fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:47:27.927429 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-fe7105a0-46da-4b22-84d0-50684711c5fc None None] MFA Rules not processed for user `6980cb57558a43d2b384c7b561d7af70`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:47:27.928691 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-fe7105a0-46da-4b22-84d0-50684711c5fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:47:27.981840 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:47:28.026704 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 663/1316] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:47:27 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:47:28.329174 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658f7863-24f9-4727-99fc-57c4fb62da4a None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:47:28.329501 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658f7863-24f9-4727-99fc-57c4fb62da4a None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:47:28.329761 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658f7863-24f9-4727-99fc-57c4fb62da4a None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:47:28.331108 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 654/1317] 10.222.0.52 () {54 vars in 964 bytes} [Sun Sep 25 00:47:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:47:28.338780 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5df7bf-b9c7-420f-8459-eef408ce1b6f None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:47:28.339054 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5df7bf-b9c7-420f-8459-eef408ce1b6f None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:47:28.339317 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5df7bf-b9c7-420f-8459-eef408ce1b6f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:47:28.421440 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-3b5df7bf-b9c7-420f-8459-eef408ce1b6f None None] MFA Rules not processed for user `20c274e96dec478b98ef51f80ea893b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:47:28.422644 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-3b5df7bf-b9c7-420f-8459-eef408ce1b6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:47:28.502339 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 664/1318] 10.222.0.52 () {58 vars in 1061 bytes} [Sun Sep 25 00:47:28 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:47:28.511885 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-658f7863-24f9-4727-99fc-57c4fb62da4a None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:47:28.513458 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-658f7863-24f9-4727-99fc-57c4fb62da4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:47:28.523967 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:47:28.624185 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-658f7863-24f9-4727-99fc-57c4fb62da4a None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:47:28.698314 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1061db9c-800e-4d55-84c4-1723ed4810a2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:47:28.698912 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1061db9c-800e-4d55-84c4-1723ed4810a2 service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:47:28.699185 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1061db9c-800e-4d55-84c4-1723ed4810a2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:47:28.699449 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1061db9c-800e-4d55-84c4-1723ed4810a2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:47:28.701046 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-1061db9c-800e-4d55-84c4-1723ed4810a2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:47:28.764719 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:47:28.811788 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1061db9c-800e-4d55-84c4-1723ed4810a2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:47:28.813206 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1061db9c-800e-4d55-84c4-1723ed4810a2 service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:47:28.831540 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 655/1319] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:47:28 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:47:31.954583 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf351ec1-4cde-4826-8f0b-579b6454aedb None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:47:31.955141 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf351ec1-4cde-4826-8f0b-579b6454aedb None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:47:31.955141 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf351ec1-4cde-4826-8f0b-579b6454aedb None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:47:31.956459 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 665/1320] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:47:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:47:31.962559 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ef3f19-99c1-4d0a-ac5d-752b85087fd6 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:47:31.963012 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ef3f19-99c1-4d0a-ac5d-752b85087fd6 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:47:31.963217 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ef3f19-99c1-4d0a-ac5d-752b85087fd6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:47:32.038045 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-18ef3f19-99c1-4d0a-ac5d-752b85087fd6 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:47:32.039782 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-18ef3f19-99c1-4d0a-ac5d-752b85087fd6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:47:32.142270 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 656/1321] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:47:31 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:47:32.432937 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf351ec1-4cde-4826-8f0b-579b6454aedb None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:47:32.434517 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-bf351ec1-4cde-4826-8f0b-579b6454aedb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:47:32.487903 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:47:32.545024 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf351ec1-4cde-4826-8f0b-579b6454aedb None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:47:32.620216 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab6a6022-d6e1-41c2-a48d-8e9d433c314f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:47:32.620816 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6a6022-d6e1-41c2-a48d-8e9d433c314f service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:47:32.621094 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6a6022-d6e1-41c2-a48d-8e9d433c314f service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:47:32.621398 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6a6022-d6e1-41c2-a48d-8e9d433c314f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:47:32.623016 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-ab6a6022-d6e1-41c2-a48d-8e9d433c314f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:47:32.720907 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:47:32.734610 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab6a6022-d6e1-41c2-a48d-8e9d433c314f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:47:32.736040 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab6a6022-d6e1-41c2-a48d-8e9d433c314f service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:47:32.754753 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 666/1322] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:47:32 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:47:35.954828 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a46d9af-1217-454f-9858-0aa1b4cbf61c None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:47:35.955366 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a46d9af-1217-454f-9858-0aa1b4cbf61c None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:47:35.955366 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a46d9af-1217-454f-9858-0aa1b4cbf61c None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:47:35.956646 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 657/1323] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:47:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:47:35.963818 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5b6b63-76a3-4c3d-a21c-6efc3e72e75c None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:47:35.964741 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5b6b63-76a3-4c3d-a21c-6efc3e72e75c None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:47:35.964741 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5b6b63-76a3-4c3d-a21c-6efc3e72e75c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:47:36.039832 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-7b5b6b63-76a3-4c3d-a21c-6efc3e72e75c None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:47:36.041049 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-7b5b6b63-76a3-4c3d-a21c-6efc3e72e75c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:47:36.148670 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 667/1324] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:47:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:47:36.438791 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a46d9af-1217-454f-9858-0aa1b4cbf61c None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:47:36.440436 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-9a46d9af-1217-454f-9858-0aa1b4cbf61c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:47:36.462580 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:47:36.560649 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a46d9af-1217-454f-9858-0aa1b4cbf61c None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:47:36.641397 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84841e61-39a7-4b7c-9204-b82a6fc9f71a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:47:36.642004 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84841e61-39a7-4b7c-9204-b82a6fc9f71a service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:47:36.642282 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84841e61-39a7-4b7c-9204-b82a6fc9f71a service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:47:36.642552 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84841e61-39a7-4b7c-9204-b82a6fc9f71a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:47:36.644173 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-84841e61-39a7-4b7c-9204-b82a6fc9f71a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:47:36.722663 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:47:36.763583 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84841e61-39a7-4b7c-9204-b82a6fc9f71a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:47:36.765015 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84841e61-39a7-4b7c-9204-b82a6fc9f71a service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:47:36.785055 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 658/1325] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:47:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:47:39.114942 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada1bd11-b9de-4657-b73e-de78f626eb68 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:47:39.115586 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada1bd11-b9de-4657-b73e-de78f626eb68 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:47:39.115586 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada1bd11-b9de-4657-b73e-de78f626eb68 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:47:39.116924 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 668/1326] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:47:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 00:47:39.124796 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0655482-aa96-430a-b40c-0c918d77838c None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:47:39.125345 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0655482-aa96-430a-b40c-0c918d77838c None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:47:39.125671 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0655482-aa96-430a-b40c-0c918d77838c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:47:39.221552 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-f0655482-aa96-430a-b40c-0c918d77838c None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:47:39.222763 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-f0655482-aa96-430a-b40c-0c918d77838c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:47:39.349534 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 659/1327] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:47:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 226 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:47:39.653977 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada1bd11-b9de-4657-b73e-de78f626eb68 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:47:39.655744 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-ada1bd11-b9de-4657-b73e-de78f626eb68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:47:39.743283 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:47:39.809477 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada1bd11-b9de-4657-b73e-de78f626eb68 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:47:39.902876 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832f9cfa-c9d6-49b4-9f83-84411da602d6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:47:39.903690 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832f9cfa-c9d6-49b4-9f83-84411da602d6 service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:47:39.904217 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832f9cfa-c9d6-49b4-9f83-84411da602d6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:47:39.904453 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832f9cfa-c9d6-49b4-9f83-84411da602d6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:47:39.906488 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-832f9cfa-c9d6-49b4-9f83-84411da602d6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:47:40.046110 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:47:40.049246 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-832f9cfa-c9d6-49b4-9f83-84411da602d6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:47:40.051315 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-832f9cfa-c9d6-49b4-9f83-84411da602d6 service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:47:40.073114 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 669/1328] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:47:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 420 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:47:42.429003 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5d82e2-e113-462a-bd3d-277bf9ef6163 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:47:42.429574 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5d82e2-e113-462a-bd3d-277bf9ef6163 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:47:42.430404 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5d82e2-e113-462a-bd3d-277bf9ef6163 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:47:42.431309 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 660/1329] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:47:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:47:42.436846 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d6c767-570c-4f4f-8752-11e2c8b80dc9 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:47:42.437325 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d6c767-570c-4f4f-8752-11e2c8b80dc9 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:47:42.437420 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d6c767-570c-4f4f-8752-11e2c8b80dc9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:47:42.510487 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-b9d6c767-570c-4f4f-8752-11e2c8b80dc9 None None] MFA Rules not processed for user `6980cb57558a43d2b384c7b561d7af70`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:47:42.511714 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-b9d6c767-570c-4f4f-8752-11e2c8b80dc9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:47:42.610976 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 670/1330] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:47:42 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:47:42.915953 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5d82e2-e113-462a-bd3d-277bf9ef6163 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:47:42.917546 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-6b5d82e2-e113-462a-bd3d-277bf9ef6163 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:47:42.950488 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:47:43.031080 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5d82e2-e113-462a-bd3d-277bf9ef6163 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:47:43.111416 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff5c705-3007-4137-af7e-7624d259ce8b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:47:43.112034 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff5c705-3007-4137-af7e-7624d259ce8b service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:47:43.112305 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff5c705-3007-4137-af7e-7624d259ce8b service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:47:43.112573 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff5c705-3007-4137-af7e-7624d259ce8b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:47:43.114212 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-1ff5c705-3007-4137-af7e-7624d259ce8b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:47:43.204997 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:47:43.238746 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ff5c705-3007-4137-af7e-7624d259ce8b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:47:43.240179 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ff5c705-3007-4137-af7e-7624d259ce8b service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:47:43.260266 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 661/1331] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:47:42 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 345 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:47:45.738804 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f96351-c2fe-4b07-a3d0-ebadd8622d95 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:47:45.739335 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f96351-c2fe-4b07-a3d0-ebadd8622d95 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:47:45.739335 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f96351-c2fe-4b07-a3d0-ebadd8622d95 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:47:45.740580 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 671/1332] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:47:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:47:45.746494 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3912752f-d440-485f-90c2-65341073740e None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:47:45.746996 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3912752f-d440-485f-90c2-65341073740e None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:47:45.747091 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3912752f-d440-485f-90c2-65341073740e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:47:45.813031 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-3912752f-d440-485f-90c2-65341073740e None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:47:45.814275 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-3912752f-d440-485f-90c2-65341073740e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:47:45.910429 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 662/1333] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:47:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:47:46.210763 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f96351-c2fe-4b07-a3d0-ebadd8622d95 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:47:46.212406 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-a0f96351-c2fe-4b07-a3d0-ebadd8622d95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:47:46.303621 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:47:46.338966 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f96351-c2fe-4b07-a3d0-ebadd8622d95 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:47:46.415137 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff8437c-c0fe-4755-a23c-d287c52d5b83 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:47:46.415733 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff8437c-c0fe-4755-a23c-d287c52d5b83 service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:47:46.416002 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff8437c-c0fe-4755-a23c-d287c52d5b83 service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:47:46.416263 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff8437c-c0fe-4755-a23c-d287c52d5b83 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:47:46.417887 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-6ff8437c-c0fe-4755-a23c-d287c52d5b83 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:47:46.527975 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ff8437c-c0fe-4755-a23c-d287c52d5b83 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:47:46.529443 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ff8437c-c0fe-4755-a23c-d287c52d5b83 service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:47:46.539827 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:47:46.548858 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 672/1334] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:47:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:47:48.784274 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69fa3fe5-798f-47da-aaac-0f54ba07bd80 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:47:48.784810 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69fa3fe5-798f-47da-aaac-0f54ba07bd80 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:47:48.784810 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69fa3fe5-798f-47da-aaac-0f54ba07bd80 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:47:48.786081 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 663/1335] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:47:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:47:48.791538 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f1a56f-51ee-4f93-aab2-d8789a1fb02e None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:47:48.791969 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f1a56f-51ee-4f93-aab2-d8789a1fb02e None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:47:48.792091 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f1a56f-51ee-4f93-aab2-d8789a1fb02e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:47:48.858261 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-c1f1a56f-51ee-4f93-aab2-d8789a1fb02e None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:47:48.859751 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-c1f1a56f-51ee-4f93-aab2-d8789a1fb02e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:47:48.956726 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 673/1336] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:47:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:47:49.243586 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69fa3fe5-798f-47da-aaac-0f54ba07bd80 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:47:49.245118 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-69fa3fe5-798f-47da-aaac-0f54ba07bd80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:47:49.286160 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:47:49.350397 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69fa3fe5-798f-47da-aaac-0f54ba07bd80 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:47:49.433113 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a79afc74-b127-44a3-a17d-f8b683ea7076 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:47:49.433743 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79afc74-b127-44a3-a17d-f8b683ea7076 service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:47:49.434017 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79afc74-b127-44a3-a17d-f8b683ea7076 service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:47:49.434285 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79afc74-b127-44a3-a17d-f8b683ea7076 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:47:49.435880 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-a79afc74-b127-44a3-a17d-f8b683ea7076 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:47:49.522612 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:47:49.548836 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a79afc74-b127-44a3-a17d-f8b683ea7076 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:47:49.550300 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a79afc74-b127-44a3-a17d-f8b683ea7076 service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:47:49.569640 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 664/1337] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:47:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 327 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 00:47:52.077078 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10158591-66a8-442a-9959-c57380e4c98d None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:47:52.077718 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10158591-66a8-442a-9959-c57380e4c98d None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:47:52.077718 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10158591-66a8-442a-9959-c57380e4c98d None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:47:52.078960 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 674/1338] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:47:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:47:52.084085 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0611a62-59b6-4da4-8ece-9feeb9f15281 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:47:52.085137 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0611a62-59b6-4da4-8ece-9feeb9f15281 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:47:52.085670 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0611a62-59b6-4da4-8ece-9feeb9f15281 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:47:52.152292 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-c0611a62-59b6-4da4-8ece-9feeb9f15281 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:47:52.153528 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-c0611a62-59b6-4da4-8ece-9feeb9f15281 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:47:52.251725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 665/1339] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:47:52 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:47:52.541422 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10158591-66a8-442a-9959-c57380e4c98d None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:47:52.543037 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-10158591-66a8-442a-9959-c57380e4c98d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:47:52.625660 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:47:52.655557 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10158591-66a8-442a-9959-c57380e4c98d None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:47:52.731925 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd6672b-c59f-492f-ad59-5a4a28ffe125 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:47:52.732529 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd6672b-c59f-492f-ad59-5a4a28ffe125 service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:47:52.732805 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd6672b-c59f-492f-ad59-5a4a28ffe125 service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:47:52.733071 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd6672b-c59f-492f-ad59-5a4a28ffe125 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:47:52.734966 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-6bd6672b-c59f-492f-ad59-5a4a28ffe125 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:47:52.844025 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bd6672b-c59f-492f-ad59-5a4a28ffe125 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:47:52.845386 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bd6672b-c59f-492f-ad59-5a4a28ffe125 service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:47:52.863314 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 675/1340] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:47:52 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:47:57.208363 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ce8263-5798-48ec-b998-4b962e62a7f2 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:47:57.208967 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ce8263-5798-48ec-b998-4b962e62a7f2 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:47:57.208967 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ce8263-5798-48ec-b998-4b962e62a7f2 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:47:57.210957 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 666/1341] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:47:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:47:57.216258 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507ad103-b6ca-4291-be82-c522d53a135f None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:47:57.216723 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507ad103-b6ca-4291-be82-c522d53a135f None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:47:57.216815 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507ad103-b6ca-4291-be82-c522d53a135f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:47:57.223336 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:47:57.294307 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-507ad103-b6ca-4291-be82-c522d53a135f None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:47:57.295591 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-507ad103-b6ca-4291-be82-c522d53a135f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:47:57.404614 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 676/1342] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:47:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:47:57.694400 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ce8263-5798-48ec-b998-4b962e62a7f2 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:47:57.696175 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-61ce8263-5798-48ec-b998-4b962e62a7f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:47:57.780818 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:47:57.844798 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ce8263-5798-48ec-b998-4b962e62a7f2 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:47:57.931000 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17b620a-01c4-4e32-8648-bb73e8844af9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:47:57.931793 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17b620a-01c4-4e32-8648-bb73e8844af9 service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:47:57.932132 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17b620a-01c4-4e32-8648-bb73e8844af9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:47:57.932491 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17b620a-01c4-4e32-8648-bb73e8844af9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:47:57.934337 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-c17b620a-01c4-4e32-8648-bb73e8844af9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:47:58.041288 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:47:58.060273 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c17b620a-01c4-4e32-8648-bb73e8844af9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:47:58.061749 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c17b620a-01c4-4e32-8648-bb73e8844af9 service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:47:58.081567 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 667/1343] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:47:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 388 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:48:00.915935 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5dc69e-e4de-4d55-90f9-ebf29f5abfb7 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:00.916490 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5dc69e-e4de-4d55-90f9-ebf29f5abfb7 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:00.916490 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5dc69e-e4de-4d55-90f9-ebf29f5abfb7 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:00.917766 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 677/1344] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:48:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:48:00.923550 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0dbee50-2c00-4cc2-8137-a5bec9272022 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:00.924117 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0dbee50-2c00-4cc2-8137-a5bec9272022 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:00.924220 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0dbee50-2c00-4cc2-8137-a5bec9272022 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:01.002407 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-e0dbee50-2c00-4cc2-8137-a5bec9272022 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:48:01.003629 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-e0dbee50-2c00-4cc2-8137-a5bec9272022 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:01.119529 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 668/1345] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:48:00 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:48:01.409398 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c5dc69e-e4de-4d55-90f9-ebf29f5abfb7 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:48:01.411010 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-7c5dc69e-e4de-4d55-90f9-ebf29f5abfb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:01.511847 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:48:01.530291 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c5dc69e-e4de-4d55-90f9-ebf29f5abfb7 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:48:01.607788 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-793bd57e-4c49-47f3-96b0-ef1f55721b8a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:48:01.608392 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-793bd57e-4c49-47f3-96b0-ef1f55721b8a service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:01.608665 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-793bd57e-4c49-47f3-96b0-ef1f55721b8a service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:01.608929 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-793bd57e-4c49-47f3-96b0-ef1f55721b8a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:01.610814 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-793bd57e-4c49-47f3-96b0-ef1f55721b8a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:01.728280 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-793bd57e-4c49-47f3-96b0-ef1f55721b8a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:48:01.729736 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-793bd57e-4c49-47f3-96b0-ef1f55721b8a service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:48:01.748003 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 678/1346] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:48:01 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:48:06.206658 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19826da7-f214-47ce-8712-5fac44b07717 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:06.207229 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19826da7-f214-47ce-8712-5fac44b07717 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:06.207229 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19826da7-f214-47ce-8712-5fac44b07717 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:06.209305 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 669/1347] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:48:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:48:06.214939 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7abd18-6e7a-44ce-a516-457ac4675e0d None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:06.215450 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7abd18-6e7a-44ce-a516-457ac4675e0d None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:06.215450 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7abd18-6e7a-44ce-a516-457ac4675e0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:06.232376 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:48:06.304618 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-8b7abd18-6e7a-44ce-a516-457ac4675e0d None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:48:06.305917 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-8b7abd18-6e7a-44ce-a516-457ac4675e0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:06.418781 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 679/1348] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:48:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:48:06.708991 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19826da7-f214-47ce-8712-5fac44b07717 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:48:06.711192 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-19826da7-f214-47ce-8712-5fac44b07717 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:06.772806 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:48:06.821519 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19826da7-f214-47ce-8712-5fac44b07717 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:48:06.896860 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-354158e2-9627-47fc-872c-a383729a9697 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:48:06.897809 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354158e2-9627-47fc-872c-a383729a9697 service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:06.898086 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354158e2-9627-47fc-872c-a383729a9697 service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:06.898356 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354158e2-9627-47fc-872c-a383729a9697 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:06.899960 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-354158e2-9627-47fc-872c-a383729a9697 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:07.014005 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:48:07.018689 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-354158e2-9627-47fc-872c-a383729a9697 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:48:07.020249 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-354158e2-9627-47fc-872c-a383729a9697 service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:48:07.040899 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 670/1349] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:48:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 333 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:48:12.246053 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4155a9-1bc0-478e-883e-bb33d0f19e9a None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:12.246808 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4155a9-1bc0-478e-883e-bb33d0f19e9a None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:12.246808 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4155a9-1bc0-478e-883e-bb33d0f19e9a None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:12.248016 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 680/1350] 10.222.0.52 () {54 vars in 921 bytes} [Sun Sep 25 00:48:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:48:12.259629 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d3a09d-4daf-4a69-b70b-5554889fd66b None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:12.260118 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d3a09d-4daf-4a69-b70b-5554889fd66b None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:12.260226 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d3a09d-4daf-4a69-b70b-5554889fd66b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:12.400175 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-14d3a09d-4daf-4a69-b70b-5554889fd66b None None] MFA Rules not processed for user `11a80b55dcd444df901c8bf80f2a2c26`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:48:12.401417 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-14d3a09d-4daf-4a69-b70b-5554889fd66b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:12.531934 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 671/1351] 10.222.0.52 () {58 vars in 1018 bytes} [Sun Sep 25 00:48:12 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 278 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:48:16.417167 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=59 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:48:16.733748 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069c89ce-b466-49f0-82d4-ce7edd69dd1c None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:16.734113 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069c89ce-b466-49f0-82d4-ce7edd69dd1c None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:16.734343 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069c89ce-b466-49f0-82d4-ce7edd69dd1c None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:16.736105 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 681/1352] 10.222.0.52 () {54 vars in 921 bytes} [Sun Sep 25 00:48:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:48:16.742168 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe828e35-d393-4b6d-9fbc-37d2a9cd3e6a None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:16.742652 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe828e35-d393-4b6d-9fbc-37d2a9cd3e6a None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:16.742733 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe828e35-d393-4b6d-9fbc-37d2a9cd3e6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:16.801758 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:48:16.831806 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-fe828e35-d393-4b6d-9fbc-37d2a9cd3e6a None None] MFA Rules not processed for user `11a80b55dcd444df901c8bf80f2a2c26`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:48:16.833026 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-fe828e35-d393-4b6d-9fbc-37d2a9cd3e6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:16.950693 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 672/1353] 10.222.0.52 () {58 vars in 1018 bytes} [Sun Sep 25 00:48:16 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:48:18.431389 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8706baae-f195-44e3-b204-c4dd762dafd0 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:18.431914 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8706baae-f195-44e3-b204-c4dd762dafd0 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:18.431914 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8706baae-f195-44e3-b204-c4dd762dafd0 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:18.433568 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 682/1354] 10.222.0.52 () {54 vars in 921 bytes} [Sun Sep 25 00:48:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:48:18.439663 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-379d7725-312f-4f26-86f6-74f33e47e4f7 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:18.440245 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-379d7725-312f-4f26-86f6-74f33e47e4f7 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:18.440245 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-379d7725-312f-4f26-86f6-74f33e47e4f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:18.532413 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-379d7725-312f-4f26-86f6-74f33e47e4f7 None None] MFA Rules not processed for user `11a80b55dcd444df901c8bf80f2a2c26`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:48:18.533695 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-379d7725-312f-4f26-86f6-74f33e47e4f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:18.584120 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:48:18.657276 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 673/1355] 10.222.0.52 () {58 vars in 1018 bytes} [Sun Sep 25 00:48:18 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 219 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:48:22.391503 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b9e4aa-724a-4b47-91a7-f0f6cd7da84a None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:22.392673 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b9e4aa-724a-4b47-91a7-f0f6cd7da84a None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:22.393393 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b9e4aa-724a-4b47-91a7-f0f6cd7da84a None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:22.398532 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 683/1356] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:48:22 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:48:22.408051 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c0948f-847a-45b7-9613-4243d2a29198 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:22.410396 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c0948f-847a-45b7-9613-4243d2a29198 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:22.411080 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c0948f-847a-45b7-9613-4243d2a29198 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:22.539931 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-42c0948f-847a-45b7-9613-4243d2a29198 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:48:22.541081 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-42c0948f-847a-45b7-9613-4243d2a29198 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:22.701090 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 674/1357] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:48:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 296 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:48:23.013089 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3223d053-d9e3-42aa-8be2-f3f3fd9197dc None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:23.014433 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3223d053-d9e3-42aa-8be2-f3f3fd9197dc None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:23.015240 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3223d053-d9e3-42aa-8be2-f3f3fd9197dc None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:23.019023 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 684/1358] 10.222.0.52 () {54 vars in 961 bytes} [Sun Sep 25 00:48:23 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:48:23.031074 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b4a9dd-8d95-400b-ad86-3554c7e300ce None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:23.032044 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b4a9dd-8d95-400b-ad86-3554c7e300ce None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:23.032938 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b4a9dd-8d95-400b-ad86-3554c7e300ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:23.064082 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:48:23.198215 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-79b4a9dd-8d95-400b-ad86-3554c7e300ce None None] MFA Rules not processed for user `f142988a4dd14a5f96eacea45c046f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:48:23.200088 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-79b4a9dd-8d95-400b-ad86-3554c7e300ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:23.444915 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 675/1359] 10.222.0.52 () {58 vars in 1058 bytes} [Sun Sep 25 00:48:23 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 416 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:48:23.458947 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3223d053-d9e3-42aa-8be2-f3f3fd9197dc None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:48:23.463672 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-3223d053-d9e3-42aa-8be2-f3f3fd9197dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:23.769477 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3223d053-d9e3-42aa-8be2-f3f3fd9197dc None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:48:23.915465 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:48:23.967400 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a03a454-6aaa-4da1-b0e7-71765f6d55a9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f142988a4dd14a5f96eacea45c046f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:48:23.968184 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a03a454-6aaa-4da1-b0e7-71765f6d55a9 service nova] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:23.968560 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a03a454-6aaa-4da1-b0e7-71765f6d55a9 service nova] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:23.968930 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a03a454-6aaa-4da1-b0e7-71765f6d55a9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:23.970708 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-6a03a454-6aaa-4da1-b0e7-71765f6d55a9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:24.101286 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a03a454-6aaa-4da1-b0e7-71765f6d55a9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:48:24.103075 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a03a454-6aaa-4da1-b0e7-71765f6d55a9 service nova] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:48:24.133560 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 685/1360] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:48:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 676 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:48:26.642193 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d797d5cb-493c-4aaa-8831-44183c36f08b None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:26.643239 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d797d5cb-493c-4aaa-8831-44183c36f08b None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:26.643239 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d797d5cb-493c-4aaa-8831-44183c36f08b None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:26.649322 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 676/1361] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:48:26 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:48:26.654984 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4e9a0a-b2ee-4300-9d07-c7ff2f428901 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:26.655898 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4e9a0a-b2ee-4300-9d07-c7ff2f428901 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:26.656056 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4e9a0a-b2ee-4300-9d07-c7ff2f428901 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:26.828697 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-fd4e9a0a-b2ee-4300-9d07-c7ff2f428901 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:48:26.828697 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-fd4e9a0a-b2ee-4300-9d07-c7ff2f428901 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:26.900920 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:48:27.041345 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 686/1362] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:48:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 389 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:48:27.348787 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d797d5cb-493c-4aaa-8831-44183c36f08b None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:48:27.352241 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-d797d5cb-493c-4aaa-8831-44183c36f08b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:27.491869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:48:27.653693 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d797d5cb-493c-4aaa-8831-44183c36f08b None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:48:27.839389 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5533f2da-6765-4cd6-baa4-1ecb4c3c548a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f142988a4dd14a5f96eacea45c046f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:48:27.840004 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5533f2da-6765-4cd6-baa4-1ecb4c3c548a service nova] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:27.840277 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5533f2da-6765-4cd6-baa4-1ecb4c3c548a service nova] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:27.840540 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5533f2da-6765-4cd6-baa4-1ecb4c3c548a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:27.842365 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-5533f2da-6765-4cd6-baa4-1ecb4c3c548a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:27.874482 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:48:27.968126 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5533f2da-6765-4cd6-baa4-1ecb4c3c548a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:48:27.970838 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5533f2da-6765-4cd6-baa4-1ecb4c3c548a service nova] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:48:28.013905 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 677/1363] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:48:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 667 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:48:31.140531 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2737c8-b332-4089-ba2a-861e3c806306 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:31.141466 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2737c8-b332-4089-ba2a-861e3c806306 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:31.141953 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2737c8-b332-4089-ba2a-861e3c806306 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:31.149397 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 687/1364] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:48:31 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:48:31.151485 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61583705-5ce1-45bc-ab2e-12da91a22d62 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:31.152503 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61583705-5ce1-45bc-ab2e-12da91a22d62 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:31.152796 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61583705-5ce1-45bc-ab2e-12da91a22d62 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:31.243991 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-61583705-5ce1-45bc-ab2e-12da91a22d62 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:48:31.245234 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-61583705-5ce1-45bc-ab2e-12da91a22d62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:31.451762 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:48:31.469803 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 678/1365] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:48:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 319 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:48:31.824287 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6db5c24-6dfd-4ad8-8ce4-7126e4782152 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:31.824587 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6db5c24-6dfd-4ad8-8ce4-7126e4782152 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:31.824826 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6db5c24-6dfd-4ad8-8ce4-7126e4782152 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:31.826647 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 688/1366] 10.222.0.52 () {54 vars in 961 bytes} [Sun Sep 25 00:48:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:48:31.835382 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c580f62a-9a1b-4292-a2a3-4335989bf81c None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:31.836065 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c580f62a-9a1b-4292-a2a3-4335989bf81c None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:31.836483 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c580f62a-9a1b-4292-a2a3-4335989bf81c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:31.939606 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-c580f62a-9a1b-4292-a2a3-4335989bf81c None None] MFA Rules not processed for user `f142988a4dd14a5f96eacea45c046f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:48:31.940911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-c580f62a-9a1b-4292-a2a3-4335989bf81c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:32.100935 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 679/1367] 10.222.0.52 () {58 vars in 1058 bytes} [Sun Sep 25 00:48:31 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 267 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:48:32.116881 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6db5c24-6dfd-4ad8-8ce4-7126e4782152 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:48:32.119912 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-c6db5c24-6dfd-4ad8-8ce4-7126e4782152 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:32.333687 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3917cfc8-a4f6-4d14-b00a-13823d6bc081 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:32.336027 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3917cfc8-a4f6-4d14-b00a-13823d6bc081 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:32.336027 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3917cfc8-a4f6-4d14-b00a-13823d6bc081 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:32.337740 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 680/1368] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:48:32 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 00:48:32.338336 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6db5c24-6dfd-4ad8-8ce4-7126e4782152 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:48:32.344942 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997789ef-b452-4977-af87-618a6f3f1d32 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:32.345402 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997789ef-b452-4977-af87-618a6f3f1d32 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:32.345783 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997789ef-b452-4977-af87-618a6f3f1d32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:32.368904 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:48:32.436208 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:48:32.449890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-997789ef-b452-4977-af87-618a6f3f1d32 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:48:32.451353 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-997789ef-b452-4977-af87-618a6f3f1d32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:32.558369 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adab9915-cf90-4a53-873d-6764ac1e40d3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f142988a4dd14a5f96eacea45c046f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:48:32.559123 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adab9915-cf90-4a53-873d-6764ac1e40d3 service nova] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:32.559467 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adab9915-cf90-4a53-873d-6764ac1e40d3 service nova] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:32.559877 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adab9915-cf90-4a53-873d-6764ac1e40d3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:32.561619 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-adab9915-cf90-4a53-873d-6764ac1e40d3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:32.628024 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 681/1369] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:48:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 284 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 00:48:32.684489 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adab9915-cf90-4a53-873d-6764ac1e40d3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:48:32.686285 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adab9915-cf90-4a53-873d-6764ac1e40d3 service nova] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:48:32.709626 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 689/1370] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:48:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 594 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:48:32.961763 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21f5ada5-d0aa-45da-859e-31b02f496910 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:32.962064 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21f5ada5-d0aa-45da-859e-31b02f496910 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:32.962321 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21f5ada5-d0aa-45da-859e-31b02f496910 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:32.963972 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 682/1371] 10.222.0.52 () {54 vars in 963 bytes} [Sun Sep 25 00:48:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:48:32.969532 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510dca78-98ee-4dc6-96ea-aed3efaba945 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:32.969876 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510dca78-98ee-4dc6-96ea-aed3efaba945 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:32.970071 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510dca78-98ee-4dc6-96ea-aed3efaba945 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:33.011072 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:48:33.087861 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-510dca78-98ee-4dc6-96ea-aed3efaba945 None None] MFA Rules not processed for user `944f10f38ee2428a85599f6a714ac806`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:48:33.089509 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-510dca78-98ee-4dc6-96ea-aed3efaba945 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:33.193587 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 690/1372] 10.222.0.52 () {58 vars in 1060 bytes} [Sun Sep 25 00:48:32 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 225 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:48:33.201626 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21f5ada5-d0aa-45da-859e-31b02f496910 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:48:33.203289 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-21f5ada5-d0aa-45da-859e-31b02f496910 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:33.340815 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21f5ada5-d0aa-45da-859e-31b02f496910 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:48:33.388108 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:48:33.465376 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aede199f-6143-4e42-8b37-8033661b556c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:48:33.466942 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aede199f-6143-4e42-8b37-8033661b556c service cinder] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:33.467720 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aede199f-6143-4e42-8b37-8033661b556c service cinder] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:33.468488 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aede199f-6143-4e42-8b37-8033661b556c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:33.472108 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-aede199f-6143-4e42-8b37-8033661b556c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:33.727076 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aede199f-6143-4e42-8b37-8033661b556c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:48:33.731334 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aede199f-6143-4e42-8b37-8033661b556c service cinder] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:48:33.765385 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 683/1373] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:48:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 559 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:48:35.294799 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33927b72-c96c-43ee-be14-d0d9e4eb5336 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:35.295640 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33927b72-c96c-43ee-be14-d0d9e4eb5336 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:35.295640 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33927b72-c96c-43ee-be14-d0d9e4eb5336 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:35.296786 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 691/1374] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:48:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 00:48:35.304072 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb99647-c323-49bc-847c-c3b2c63bdc44 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:35.304915 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb99647-c323-49bc-847c-c3b2c63bdc44 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:35.304915 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb99647-c323-49bc-847c-c3b2c63bdc44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:35.405366 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-bcb99647-c323-49bc-847c-c3b2c63bdc44 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:48:35.406626 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-bcb99647-c323-49bc-847c-c3b2c63bdc44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:35.416111 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:48:35.537003 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 684/1375] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:48:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 235 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:48:35.896143 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33927b72-c96c-43ee-be14-d0d9e4eb5336 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:48:35.901816 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-33927b72-c96c-43ee-be14-d0d9e4eb5336 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:36.071823 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:48:36.122158 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33927b72-c96c-43ee-be14-d0d9e4eb5336 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:48:36.296425 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ede90c1-f660-4d2b-895b-843afb18aac7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f142988a4dd14a5f96eacea45c046f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:48:36.297315 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ede90c1-f660-4d2b-895b-843afb18aac7 service nova] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:36.298969 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ede90c1-f660-4d2b-895b-843afb18aac7 service nova] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:36.298969 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ede90c1-f660-4d2b-895b-843afb18aac7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:36.299697 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-3ede90c1-f660-4d2b-895b-843afb18aac7 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:36.412351 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:48:36.438070 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ede90c1-f660-4d2b-895b-843afb18aac7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:48:36.439705 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ede90c1-f660-4d2b-895b-843afb18aac7 service nova] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:48:36.465590 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 692/1376] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:48:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 570 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:48:38.922213 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc65902-0035-4309-9b1d-498020028dac None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:38.922750 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc65902-0035-4309-9b1d-498020028dac None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:38.922750 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc65902-0035-4309-9b1d-498020028dac None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:38.924188 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 685/1377] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:48:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:48:38.930923 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77930e45-dc1c-4e2a-b11b-c4a3a995aefc None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:38.931475 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77930e45-dc1c-4e2a-b11b-c4a3a995aefc None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:38.931853 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77930e45-dc1c-4e2a-b11b-c4a3a995aefc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:39.023122 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-77930e45-dc1c-4e2a-b11b-c4a3a995aefc None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:48:39.024569 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-77930e45-dc1c-4e2a-b11b-c4a3a995aefc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:39.157299 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 693/1378] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:48:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 225 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:48:39.469018 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc65902-0035-4309-9b1d-498020028dac None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:48:39.470848 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-1dc65902-0035-4309-9b1d-498020028dac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:39.664387 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:48:39.677414 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc65902-0035-4309-9b1d-498020028dac None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:48:39.821672 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b1346f2-f8ff-475f-a3aa-13d62d48093d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f142988a4dd14a5f96eacea45c046f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:48:39.822299 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1346f2-f8ff-475f-a3aa-13d62d48093d service nova] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:39.822560 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1346f2-f8ff-475f-a3aa-13d62d48093d service nova] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:39.822824 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1346f2-f8ff-475f-a3aa-13d62d48093d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:39.824465 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-2b1346f2-f8ff-475f-a3aa-13d62d48093d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:39.956801 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b1346f2-f8ff-475f-a3aa-13d62d48093d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:48:39.958573 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b1346f2-f8ff-475f-a3aa-13d62d48093d service nova] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:48:39.972989 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:48:39.989319 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 686/1379] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:48:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 521 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 00:48:42.551292 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca38e213-7585-4b56-a20b-4576bb32e0f4 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:42.552112 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca38e213-7585-4b56-a20b-4576bb32e0f4 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:42.552112 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca38e213-7585-4b56-a20b-4576bb32e0f4 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:42.553933 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 694/1380] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:48:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:48:42.559952 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a16d53c6-63eb-4f9f-8732-ac51bf09b330 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:42.560529 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a16d53c6-63eb-4f9f-8732-ac51bf09b330 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:42.560613 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a16d53c6-63eb-4f9f-8732-ac51bf09b330 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:42.694878 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-a16d53c6-63eb-4f9f-8732-ac51bf09b330 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:48:42.696255 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-a16d53c6-63eb-4f9f-8732-ac51bf09b330 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:42.833135 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 687/1381] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:48:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 274 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:48:43.149121 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca38e213-7585-4b56-a20b-4576bb32e0f4 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:48:43.150851 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-ca38e213-7585-4b56-a20b-4576bb32e0f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:43.232142 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:48:43.356300 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca38e213-7585-4b56-a20b-4576bb32e0f4 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:48:43.517108 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fec61d7-d4b5-4376-b2df-1c2d51815c9c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f142988a4dd14a5f96eacea45c046f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:48:43.517752 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fec61d7-d4b5-4376-b2df-1c2d51815c9c service nova] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:43.518030 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fec61d7-d4b5-4376-b2df-1c2d51815c9c service nova] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:43.518295 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fec61d7-d4b5-4376-b2df-1c2d51815c9c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:43.519936 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-2fec61d7-d4b5-4376-b2df-1c2d51815c9c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:43.545562 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:48:43.646481 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fec61d7-d4b5-4376-b2df-1c2d51815c9c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:48:43.648929 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fec61d7-d4b5-4376-b2df-1c2d51815c9c service nova] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:48:43.672104 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 695/1382] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:48:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 523 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:48:46.104345 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbc71cd-351b-4f9b-860d-9c4f470cc71d None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:46.104959 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbc71cd-351b-4f9b-860d-9c4f470cc71d None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:46.105092 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbc71cd-351b-4f9b-860d-9c4f470cc71d None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:46.107038 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 688/1383] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:48:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 00:48:46.116978 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893ea336-8932-44ff-a4ce-1d07b67d5c4d None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:46.117858 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893ea336-8932-44ff-a4ce-1d07b67d5c4d None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:46.118088 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893ea336-8932-44ff-a4ce-1d07b67d5c4d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:46.222272 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-893ea336-8932-44ff-a4ce-1d07b67d5c4d None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:48:46.226365 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-893ea336-8932-44ff-a4ce-1d07b67d5c4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:46.367317 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:48:46.375053 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 696/1384] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:48:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 260 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:48:46.696136 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbc71cd-351b-4f9b-860d-9c4f470cc71d None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:48:46.697779 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-8fbc71cd-351b-4f9b-860d-9c4f470cc71d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:46.818692 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:48:46.922108 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbc71cd-351b-4f9b-860d-9c4f470cc71d None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:48:47.116668 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e2b0698-5299-43d4-bf61-0ef4fff75a1b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f142988a4dd14a5f96eacea45c046f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:48:47.117891 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2b0698-5299-43d4-bf61-0ef4fff75a1b service nova] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:47.117891 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2b0698-5299-43d4-bf61-0ef4fff75a1b service nova] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:47.118193 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2b0698-5299-43d4-bf61-0ef4fff75a1b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:47.121170 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-8e2b0698-5299-43d4-bf61-0ef4fff75a1b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:47.178215 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:48:47.257004 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e2b0698-5299-43d4-bf61-0ef4fff75a1b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:48:47.259098 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e2b0698-5299-43d4-bf61-0ef4fff75a1b service nova] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:48:47.292357 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 689/1385] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:48:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 597 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:48:49.679865 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4ae040-d35c-471a-ac07-d6f25725c32d None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:49.680412 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4ae040-d35c-471a-ac07-d6f25725c32d None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:49.680531 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4ae040-d35c-471a-ac07-d6f25725c32d None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:49.682315 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 697/1386] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:48:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:48:49.688014 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0408864-83a5-44e4-b959-26a8c5dbf2d1 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:49.688516 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0408864-83a5-44e4-b959-26a8c5dbf2d1 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:49.688632 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0408864-83a5-44e4-b959-26a8c5dbf2d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:49.772610 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-f0408864-83a5-44e4-b959-26a8c5dbf2d1 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:48:49.773892 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-f0408864-83a5-44e4-b959-26a8c5dbf2d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:49.891215 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 690/1387] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:48:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:48:50.215374 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f4ae040-d35c-471a-ac07-d6f25725c32d None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:48:50.216909 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-4f4ae040-d35c-471a-ac07-d6f25725c32d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:50.398449 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f4ae040-d35c-471a-ac07-d6f25725c32d None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:48:50.511334 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:48:50.576662 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b0d464-c2fe-42e4-819f-6818d181d1fc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f142988a4dd14a5f96eacea45c046f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:48:50.576662 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b0d464-c2fe-42e4-819f-6818d181d1fc service nova] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:50.576662 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b0d464-c2fe-42e4-819f-6818d181d1fc service nova] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:50.576662 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b0d464-c2fe-42e4-819f-6818d181d1fc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:50.577235 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-69b0d464-c2fe-42e4-819f-6818d181d1fc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:50.578033 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476395d4-1826-4ac4-8f69-df5b0b517a93 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:50.578406 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476395d4-1826-4ac4-8f69-df5b0b517a93 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:50.578714 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476395d4-1826-4ac4-8f69-df5b0b517a93 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:50.580314 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 691/1388] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:48:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 00:48:50.587384 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37d73ea-8171-4d48-a3b0-7aa6aba41091 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:50.587688 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37d73ea-8171-4d48-a3b0-7aa6aba41091 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:50.587921 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37d73ea-8171-4d48-a3b0-7aa6aba41091 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:50.608395 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:48:50.692921 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-b37d73ea-8171-4d48-a3b0-7aa6aba41091 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:48:50.695551 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-b37d73ea-8171-4d48-a3b0-7aa6aba41091 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:50.701993 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69b0d464-c2fe-42e4-819f-6818d181d1fc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:48:50.703738 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69b0d464-c2fe-42e4-819f-6818d181d1fc service nova] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:48:50.730109 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 698/1389] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:48:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 523 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:48:50.824822 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 692/1390] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:48:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 238 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 00:48:51.154652 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eafc564e-4b1e-414b-9e3a-3f1952a15069 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:51.154938 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eafc564e-4b1e-414b-9e3a-3f1952a15069 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:51.155203 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eafc564e-4b1e-414b-9e3a-3f1952a15069 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:51.156524 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 699/1391] 10.222.0.52 () {54 vars in 962 bytes} [Sun Sep 25 00:48:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:48:51.162978 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a778d7b0-3eff-40f6-97fc-66cc73a2fd40 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:51.163290 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a778d7b0-3eff-40f6-97fc-66cc73a2fd40 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:51.163580 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a778d7b0-3eff-40f6-97fc-66cc73a2fd40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:51.260385 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-a778d7b0-3eff-40f6-97fc-66cc73a2fd40 None None] MFA Rules not processed for user `4ebfc0abf1b842459a44ca9324252688`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:48:51.261811 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-a778d7b0-3eff-40f6-97fc-66cc73a2fd40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:51.306437 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:48:51.376343 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 693/1392] 10.222.0.52 () {58 vars in 1059 bytes} [Sun Sep 25 00:48:51 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 00:48:51.389848 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eafc564e-4b1e-414b-9e3a-3f1952a15069 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:48:51.392678 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-eafc564e-4b1e-414b-9e3a-3f1952a15069 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:51.510592 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:48:51.536262 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eafc564e-4b1e-414b-9e3a-3f1952a15069 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:48:51.631239 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-906549e8-8024-42a7-9782-4897a8078010 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4ebfc0abf1b842459a44ca9324252688', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:48:51.631902 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906549e8-8024-42a7-9782-4897a8078010 service glance] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:51.632174 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906549e8-8024-42a7-9782-4897a8078010 service glance] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:51.632438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906549e8-8024-42a7-9782-4897a8078010 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:51.634144 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-906549e8-8024-42a7-9782-4897a8078010 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:51.807176 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-906549e8-8024-42a7-9782-4897a8078010 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:48:51.808683 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-906549e8-8024-42a7-9782-4897a8078010 service glance] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:48:51.830911 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 700/1393] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:48:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 442 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:48:52.703075 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ad512b-6185-4074-9718-79817dbeceb1 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:52.703738 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ad512b-6185-4074-9718-79817dbeceb1 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:52.703926 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ad512b-6185-4074-9718-79817dbeceb1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:52.817667 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-f4ad512b-6185-4074-9718-79817dbeceb1 None None] MFA Rules not processed for user `3f20ffd8df994c1db1b1e7248db5b5a3`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:48:52.818855 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-f4ad512b-6185-4074-9718-79817dbeceb1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:52.936611 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 694/1394] 10.222.0.52 () {58 vars in 1009 bytes} [Sun Sep 25 00:48:52 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 235 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:48:52.985207 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d51ece2-5596-49bc-870d-b3c5e265f4c5 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:52.986394 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d51ece2-5596-49bc-870d-b3c5e265f4c5 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:52.986615 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d51ece2-5596-49bc-870d-b3c5e265f4c5 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:52.988139 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 701/1395] 10.222.0.52 () {54 vars in 943 bytes} [Sun Sep 25 00:48:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:48:52.996592 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f80e05b-e515-4818-a5cb-344419b0a4cd None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:52.996873 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f80e05b-e515-4818-a5cb-344419b0a4cd None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:52.997161 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f80e05b-e515-4818-a5cb-344419b0a4cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:53.078015 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:48:53.101456 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-7f80e05b-e515-4818-a5cb-344419b0a4cd None None] MFA Rules not processed for user `da606b3906dd4171bd76fd281e03e258`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:48:53.102847 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-7f80e05b-e515-4818-a5cb-344419b0a4cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:53.203895 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69d776e-9d63-476c-8751-915b2dc8b764 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:53.204163 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69d776e-9d63-476c-8751-915b2dc8b764 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:53.204578 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69d776e-9d63-476c-8751-915b2dc8b764 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:53.206412 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 702/1396] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:48:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 00:48:53.220498 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f45cf09-835c-430b-8a36-92c71ab77124 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:53.220761 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f45cf09-835c-430b-8a36-92c71ab77124 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:53.221379 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f45cf09-835c-430b-8a36-92c71ab77124 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:53.237932 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:48:53.238770 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 695/1397] 10.222.0.52 () {58 vars in 1040 bytes} [Sun Sep 25 00:48:52 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 244 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 00:48:53.249811 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f80e05b-e515-4818-a5cb-344419b0a4cd None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:48:53.261487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f80e05b-e515-4818-a5cb-344419b0a4cd None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:48:53.316119 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-3f45cf09-835c-430b-8a36-92c71ab77124 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:48:53.317440 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-3f45cf09-835c-430b-8a36-92c71ab77124 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:53.390964 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:48:53.398158 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2634ad9-879e-4c0d-bc1e-63e0c3639942 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da606b3906dd4171bd76fd281e03e258', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:48:53.398777 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2634ad9-879e-4c0d-bc1e-63e0c3639942 service swift] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:53.399050 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2634ad9-879e-4c0d-bc1e-63e0c3639942 service swift] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:53.399322 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2634ad9-879e-4c0d-bc1e-63e0c3639942 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:53.401132 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-a2634ad9-879e-4c0d-bc1e-63e0c3639942 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:53.453770 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 703/1398] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:48:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 240 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:48:53.522159 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2634ad9-879e-4c0d-bc1e-63e0c3639942 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:48:53.523814 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2634ad9-879e-4c0d-bc1e-63e0c3639942 service swift] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:48:53.533564 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 696/1399] 10.222.0.52 () {60 vars in 1384 bytes} [Sun Sep 25 00:48:53 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 285 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 25 00:48:53.766801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f45cf09-835c-430b-8a36-92c71ab77124 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:48:53.769576 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-3f45cf09-835c-430b-8a36-92c71ab77124 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:53.901068 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:48:53.948203 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f45cf09-835c-430b-8a36-92c71ab77124 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:48:54.124085 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20697fab-0e7e-461c-9d97-9567dcad303d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f142988a4dd14a5f96eacea45c046f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:48:54.124825 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20697fab-0e7e-461c-9d97-9567dcad303d service nova] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:54.125131 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20697fab-0e7e-461c-9d97-9567dcad303d service nova] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:54.125491 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20697fab-0e7e-461c-9d97-9567dcad303d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:54.127197 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-20697fab-0e7e-461c-9d97-9567dcad303d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:54.228638 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:48:54.232998 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20697fab-0e7e-461c-9d97-9567dcad303d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:48:54.235568 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20697fab-0e7e-461c-9d97-9567dcad303d service nova] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:48:54.260429 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 704/1400] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:48:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 495 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 00:48:56.523804 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485d536a-bd19-4563-b819-cfd1c157015d None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:56.524752 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485d536a-bd19-4563-b819-cfd1c157015d None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:56.525034 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485d536a-bd19-4563-b819-cfd1c157015d None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:56.526425 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 697/1401] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:48:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:48:56.532069 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e457436d-afa8-46b0-ba75-9b90f0c23cdb None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:56.532528 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e457436d-afa8-46b0-ba75-9b90f0c23cdb None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:56.532635 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e457436d-afa8-46b0-ba75-9b90f0c23cdb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:56.609720 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-e457436d-afa8-46b0-ba75-9b90f0c23cdb None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:48:56.611226 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-e457436d-afa8-46b0-ba75-9b90f0c23cdb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:56.728916 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 705/1402] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:48:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:48:57.028721 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485d536a-bd19-4563-b819-cfd1c157015d None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:48:57.030544 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-485d536a-bd19-4563-b819-cfd1c157015d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:57.201481 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:48:57.215367 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485d536a-bd19-4563-b819-cfd1c157015d None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:48:57.355172 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b38680-84d0-499b-b401-d398e4b2cbdc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f142988a4dd14a5f96eacea45c046f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:48:57.356049 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b38680-84d0-499b-b401-d398e4b2cbdc service nova] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:57.356557 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b38680-84d0-499b-b401-d398e4b2cbdc service nova] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:57.356829 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b38680-84d0-499b-b401-d398e4b2cbdc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:57.360652 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-90b38680-84d0-499b-b401-d398e4b2cbdc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:57.465646 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90b38680-84d0-499b-b401-d398e4b2cbdc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:48:57.467257 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90b38680-84d0-499b-b401-d398e4b2cbdc service nova] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:48:57.484699 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:48:57.492287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 698/1403] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:48:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 464 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:48:59.779604 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb5eee8-eac3-4f62-a237-a07b575922af None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:59.780129 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb5eee8-eac3-4f62-a237-a07b575922af None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:59.780129 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb5eee8-eac3-4f62-a237-a07b575922af None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:59.781790 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 706/1404] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:48:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:48:59.788839 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bab925-262d-45ae-bc0c-9db7a2043f19 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:48:59.789683 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bab925-262d-45ae-bc0c-9db7a2043f19 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:48:59.789956 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bab925-262d-45ae-bc0c-9db7a2043f19 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:48:59.864911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-76bab925-262d-45ae-bc0c-9db7a2043f19 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:48:59.866484 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-76bab925-262d-45ae-bc0c-9db7a2043f19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:48:59.979630 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 699/1405] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:48:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:49:00.274808 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdb5eee8-eac3-4f62-a237-a07b575922af None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:49:00.276341 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-fdb5eee8-eac3-4f62-a237-a07b575922af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:00.351055 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:49:00.442687 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdb5eee8-eac3-4f62-a237-a07b575922af None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:49:00.578391 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30308ecb-955c-4a72-922b-22cdb38f763c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f142988a4dd14a5f96eacea45c046f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:49:00.578905 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30308ecb-955c-4a72-922b-22cdb38f763c service nova] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:00.579179 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30308ecb-955c-4a72-922b-22cdb38f763c service nova] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:00.579440 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30308ecb-955c-4a72-922b-22cdb38f763c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:00.581029 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-30308ecb-955c-4a72-922b-22cdb38f763c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:00.611231 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:49:00.692845 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30308ecb-955c-4a72-922b-22cdb38f763c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:49:00.695023 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30308ecb-955c-4a72-922b-22cdb38f763c service nova] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:49:00.714242 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 707/1406] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:49:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 440 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:49:03.065957 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14898039-63f6-46b0-b55c-1df2284f633e None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:03.066547 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14898039-63f6-46b0-b55c-1df2284f633e None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:03.066547 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14898039-63f6-46b0-b55c-1df2284f633e None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:03.067833 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 700/1407] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:49:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:49:03.073740 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5074f7-dd16-4a6b-b294-9ff28f4a0fe4 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:03.074261 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5074f7-dd16-4a6b-b294-9ff28f4a0fe4 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:03.074261 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5074f7-dd16-4a6b-b294-9ff28f4a0fe4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:03.156466 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-fd5074f7-dd16-4a6b-b294-9ff28f4a0fe4 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:49:03.157747 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-fd5074f7-dd16-4a6b-b294-9ff28f4a0fe4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:03.273587 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 708/1408] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:49:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:49:03.577463 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14898039-63f6-46b0-b55c-1df2284f633e None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:49:03.579060 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-14898039-63f6-46b0-b55c-1df2284f633e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:03.671391 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:49:03.766693 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14898039-63f6-46b0-b55c-1df2284f633e None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:49:03.916918 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c6240a5-a38c-4ebc-9e02-8932c2e923e5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f142988a4dd14a5f96eacea45c046f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:49:03.916918 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6240a5-a38c-4ebc-9e02-8932c2e923e5 service nova] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:03.916918 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6240a5-a38c-4ebc-9e02-8932c2e923e5 service nova] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:03.916918 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6240a5-a38c-4ebc-9e02-8932c2e923e5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:03.918316 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-3c6240a5-a38c-4ebc-9e02-8932c2e923e5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:03.978175 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:49:04.039401 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c6240a5-a38c-4ebc-9e02-8932c2e923e5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:49:04.041196 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c6240a5-a38c-4ebc-9e02-8932c2e923e5 service nova] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:49:04.073590 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 701/1409] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:49:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 497 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:49:13.272226 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:49:14.069491 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=29 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:49:16.837913 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82e3c8a-171c-4db6-8968-e8dc05e99b6c None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:16.838456 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82e3c8a-171c-4db6-8968-e8dc05e99b6c None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:16.838456 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82e3c8a-171c-4db6-8968-e8dc05e99b6c None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:16.839736 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 709/1410] 10.222.0.52 () {54 vars in 918 bytes} [Sun Sep 25 00:49:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:49:16.845116 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d0d2bf-edce-4cb2-9e32-614340746a5e None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:16.845624 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d0d2bf-edce-4cb2-9e32-614340746a5e None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:16.845736 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d0d2bf-edce-4cb2-9e32-614340746a5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:16.930450 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-84d0d2bf-edce-4cb2-9e32-614340746a5e None None] MFA Rules not processed for user `11a80b55dcd444df901c8bf80f2a2c26`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:49:16.931668 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-84d0d2bf-edce-4cb2-9e32-614340746a5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:17.057674 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 702/1411] 10.222.0.52 () {58 vars in 1015 bytes} [Sun Sep 25 00:49:16 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:49:17.073291 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ea3099-8d5d-4eb8-8128-988310e0c5e2 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:17.073558 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ea3099-8d5d-4eb8-8128-988310e0c5e2 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:17.073824 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ea3099-8d5d-4eb8-8128-988310e0c5e2 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:17.075160 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 710/1412] 10.222.0.52 () {54 vars in 918 bytes} [Sun Sep 25 00:49:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:49:17.080459 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a2b914-ab18-4c80-8499-422015eaa5f6 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:17.080766 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a2b914-ab18-4c80-8499-422015eaa5f6 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:17.081037 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a2b914-ab18-4c80-8499-422015eaa5f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:17.161330 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:49:17.164743 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-04a2b914-ab18-4c80-8499-422015eaa5f6 None None] MFA Rules not processed for user `f142988a4dd14a5f96eacea45c046f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:49:17.166615 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-04a2b914-ab18-4c80-8499-422015eaa5f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:17.290179 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 703/1413] 10.222.0.52 () {58 vars in 1015 bytes} [Sun Sep 25 00:49:17 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:49:20.001658 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9457085-9f32-48f5-a6b7-339923f57d58 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:20.002386 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9457085-9f32-48f5-a6b7-339923f57d58 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:20.002386 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9457085-9f32-48f5-a6b7-339923f57d58 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:20.003588 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 711/1414] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 25 00:49:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:49:20.009433 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334edc92-8dc2-4181-b6d3-91710853d215 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:20.010076 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334edc92-8dc2-4181-b6d3-91710853d215 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:20.010076 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334edc92-8dc2-4181-b6d3-91710853d215 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:20.092658 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-334edc92-8dc2-4181-b6d3-91710853d215 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:49:20.093902 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-334edc92-8dc2-4181-b6d3-91710853d215 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:20.152894 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:49:20.198513 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 704/1415] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 25 00:49:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:49:20.499016 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9457085-9f32-48f5-a6b7-339923f57d58 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:49:20.501507 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-f9457085-9f32-48f5-a6b7-339923f57d58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:20.687113 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9457085-9f32-48f5-a6b7-339923f57d58 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:49:20.777093 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:49:20.822244 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ff70cc1-4006-4ff2-8f3f-0afd3fbb5994 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f142988a4dd14a5f96eacea45c046f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:49:20.822855 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff70cc1-4006-4ff2-8f3f-0afd3fbb5994 service nova] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:20.823162 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff70cc1-4006-4ff2-8f3f-0afd3fbb5994 service nova] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:20.823430 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff70cc1-4006-4ff2-8f3f-0afd3fbb5994 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:20.825069 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-0ff70cc1-4006-4ff2-8f3f-0afd3fbb5994 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:20.925286 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ff70cc1-4006-4ff2-8f3f-0afd3fbb5994 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:49:20.926898 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ff70cc1-4006-4ff2-8f3f-0afd3fbb5994 service nova] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:49:20.947401 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 712/1416] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:49:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 449 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:49:22.301993 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216a07c2-6dda-460e-bce4-0b2895903bfb None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:22.302557 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216a07c2-6dda-460e-bce4-0b2895903bfb None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:22.302557 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216a07c2-6dda-460e-bce4-0b2895903bfb None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:22.303932 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 705/1417] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 25 00:49:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 00:49:22.312015 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4390d1-a28d-4731-af87-49ba3b90a112 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:22.312584 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4390d1-a28d-4731-af87-49ba3b90a112 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:22.312584 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4390d1-a28d-4731-af87-49ba3b90a112 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:22.401644 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-0a4390d1-a28d-4731-af87-49ba3b90a112 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:49:22.403053 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-0a4390d1-a28d-4731-af87-49ba3b90a112 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:22.435043 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:49:22.547491 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 713/1418] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 25 00:49:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 237 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:49:22.710600 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-216a07c2-6dda-460e-bce4-0b2895903bfb None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:49:22.712144 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-216a07c2-6dda-460e-bce4-0b2895903bfb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:22.886507 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-216a07c2-6dda-460e-bce4-0b2895903bfb None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:49:22.931679 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:49:23.024990 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c4c2d3-8ea4-4848-9576-2cb9e044c649 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f142988a4dd14a5f96eacea45c046f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:49:23.025726 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c4c2d3-8ea4-4848-9576-2cb9e044c649 service nova] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:23.026000 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c4c2d3-8ea4-4848-9576-2cb9e044c649 service nova] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:23.026271 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c4c2d3-8ea4-4848-9576-2cb9e044c649 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:23.028109 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-55c4c2d3-8ea4-4848-9576-2cb9e044c649 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:23.142615 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55c4c2d3-8ea4-4848-9576-2cb9e044c649 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:49:23.144137 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55c4c2d3-8ea4-4848-9576-2cb9e044c649 service nova] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:49:23.165171 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 706/1419] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:49:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 455 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:49:24.472704 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d75efab-7236-4db8-8492-654ff9b83964 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:24.473544 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d75efab-7236-4db8-8492-654ff9b83964 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:24.473544 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d75efab-7236-4db8-8492-654ff9b83964 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:24.475043 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 714/1420] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 25 00:49:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:49:24.483093 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9233890-b8da-4ae5-859b-df0f73f4fb99 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:24.483971 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9233890-b8da-4ae5-859b-df0f73f4fb99 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:24.483971 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9233890-b8da-4ae5-859b-df0f73f4fb99 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:24.560603 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:49:24.588947 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-f9233890-b8da-4ae5-859b-df0f73f4fb99 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:49:24.590213 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-f9233890-b8da-4ae5-859b-df0f73f4fb99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:24.717129 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 707/1421] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 25 00:49:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 235 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 00:49:24.884041 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d75efab-7236-4db8-8492-654ff9b83964 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:49:24.885597 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-9d75efab-7236-4db8-8492-654ff9b83964 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:25.061841 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d75efab-7236-4db8-8492-654ff9b83964 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:49:25.075744 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:49:25.191215 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c981bdd8-480f-4366-aac5-88cdb29a085b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f142988a4dd14a5f96eacea45c046f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:49:25.191822 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c981bdd8-480f-4366-aac5-88cdb29a085b service nova] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:25.192090 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c981bdd8-480f-4366-aac5-88cdb29a085b service nova] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:25.192350 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c981bdd8-480f-4366-aac5-88cdb29a085b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:25.194414 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-c981bdd8-480f-4366-aac5-88cdb29a085b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:25.299582 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c981bdd8-480f-4366-aac5-88cdb29a085b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:49:25.301203 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c981bdd8-480f-4366-aac5-88cdb29a085b service nova] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:49:25.322227 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 715/1422] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:49:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 439 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:49:26.551690 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f1b17e-e339-4618-aba6-b7a69a4439a0 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:26.552392 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f1b17e-e339-4618-aba6-b7a69a4439a0 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:26.552392 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f1b17e-e339-4618-aba6-b7a69a4439a0 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:26.554087 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 708/1423] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 25 00:49:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:49:26.562408 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c083bbd7-6f70-4ac8-8147-435248d5bcaa None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:26.562955 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c083bbd7-6f70-4ac8-8147-435248d5bcaa None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:26.562955 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c083bbd7-6f70-4ac8-8147-435248d5bcaa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:26.586283 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:49:26.657392 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-c083bbd7-6f70-4ac8-8147-435248d5bcaa None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:49:26.658637 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-c083bbd7-6f70-4ac8-8147-435248d5bcaa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:26.774477 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 716/1424] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 25 00:49:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 213 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:49:26.946872 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f1b17e-e339-4618-aba6-b7a69a4439a0 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:49:26.950772 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-a5f1b17e-e339-4618-aba6-b7a69a4439a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:27.083609 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:49:27.149666 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f1b17e-e339-4618-aba6-b7a69a4439a0 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:49:27.348821 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa545649-e44b-472e-8c73-1099d423796a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f142988a4dd14a5f96eacea45c046f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:49:27.349508 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa545649-e44b-472e-8c73-1099d423796a service nova] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:27.349982 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa545649-e44b-472e-8c73-1099d423796a service nova] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:27.350608 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa545649-e44b-472e-8c73-1099d423796a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:27.353091 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-aa545649-e44b-472e-8c73-1099d423796a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:27.446311 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:49:27.461735 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa545649-e44b-472e-8c73-1099d423796a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:49:27.463326 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa545649-e44b-472e-8c73-1099d423796a service nova] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:49:27.484428 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 709/1425] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:49:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 538 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:49:28.700523 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bdd395e-930e-4460-98e1-0e0c0da88f2b None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:28.701140 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bdd395e-930e-4460-98e1-0e0c0da88f2b None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:28.701140 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bdd395e-930e-4460-98e1-0e0c0da88f2b None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:28.702511 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 717/1426] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 25 00:49:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:49:28.709980 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7a266b-67bc-44be-ae85-c4b97bcbe9e8 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:28.710662 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7a266b-67bc-44be-ae85-c4b97bcbe9e8 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:28.710662 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7a266b-67bc-44be-ae85-c4b97bcbe9e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:28.831559 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-8f7a266b-67bc-44be-ae85-c4b97bcbe9e8 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:49:28.832883 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-8f7a266b-67bc-44be-ae85-c4b97bcbe9e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:28.977427 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 710/1427] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 25 00:49:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 269 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 00:49:29.152744 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bdd395e-930e-4460-98e1-0e0c0da88f2b None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:49:29.154719 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-1bdd395e-930e-4460-98e1-0e0c0da88f2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:29.258246 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:49:29.339696 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bdd395e-930e-4460-98e1-0e0c0da88f2b None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:49:29.475842 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184e3632-5f21-4b77-bccf-a676866051d7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f142988a4dd14a5f96eacea45c046f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:49:29.476460 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184e3632-5f21-4b77-bccf-a676866051d7 service nova] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:29.476730 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184e3632-5f21-4b77-bccf-a676866051d7 service nova] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:29.477004 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184e3632-5f21-4b77-bccf-a676866051d7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:29.478835 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-184e3632-5f21-4b77-bccf-a676866051d7 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:29.538066 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:49:29.590253 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-184e3632-5f21-4b77-bccf-a676866051d7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:49:29.591861 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-184e3632-5f21-4b77-bccf-a676866051d7 service nova] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:49:29.610615 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 718/1428] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:49:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 458 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:49:30.742231 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee18493-dfa1-4bee-8498-de20b96b2a5a None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:30.743362 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee18493-dfa1-4bee-8498-de20b96b2a5a None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:30.743362 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee18493-dfa1-4bee-8498-de20b96b2a5a None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:30.744004 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 711/1429] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 25 00:49:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:49:30.750995 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ba74ba-2f7c-4013-a69f-94c35600b96e None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:30.751701 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ba74ba-2f7c-4013-a69f-94c35600b96e None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:30.751701 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ba74ba-2f7c-4013-a69f-94c35600b96e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:30.837281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-50ba74ba-2f7c-4013-a69f-94c35600b96e None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:49:30.838513 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-50ba74ba-2f7c-4013-a69f-94c35600b96e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:30.941398 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 719/1430] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 25 00:49:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:49:31.093449 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee18493-dfa1-4bee-8498-de20b96b2a5a None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:49:31.094936 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-fee18493-dfa1-4bee-8498-de20b96b2a5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:31.145638 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:49:31.246975 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee18493-dfa1-4bee-8498-de20b96b2a5a None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:49:31.363093 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be40137d-1893-436a-9b02-5742128e2ada service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f142988a4dd14a5f96eacea45c046f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:49:31.363622 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be40137d-1893-436a-9b02-5742128e2ada service nova] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:31.363975 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be40137d-1893-436a-9b02-5742128e2ada service nova] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:31.364241 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be40137d-1893-436a-9b02-5742128e2ada service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:31.365992 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-be40137d-1893-436a-9b02-5742128e2ada service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:31.373523 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:49:31.458946 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be40137d-1893-436a-9b02-5742128e2ada service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:49:31.460840 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be40137d-1893-436a-9b02-5742128e2ada service nova] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:49:31.479451 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 712/1431] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:49:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 386 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:49:32.553763 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421626dc-8d76-42d8-8ee7-18539932b6ed None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:32.554350 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421626dc-8d76-42d8-8ee7-18539932b6ed None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:32.554350 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421626dc-8d76-42d8-8ee7-18539932b6ed None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:32.555647 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 720/1432] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 25 00:49:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:49:32.562365 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66819d55-096d-47de-974d-ee7887374f9b None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:32.562889 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66819d55-096d-47de-974d-ee7887374f9b None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:32.562889 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66819d55-096d-47de-974d-ee7887374f9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:32.641639 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-66819d55-096d-47de-974d-ee7887374f9b None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:49:32.642851 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-66819d55-096d-47de-974d-ee7887374f9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:32.714283 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:49:32.746154 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 713/1433] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 25 00:49:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:49:32.908821 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-421626dc-8d76-42d8-8ee7-18539932b6ed None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:49:32.910369 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-421626dc-8d76-42d8-8ee7-18539932b6ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:32.928233 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:49:33.059203 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-421626dc-8d76-42d8-8ee7-18539932b6ed None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:49:33.147551 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:49:33.173525 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729e332d-3623-4b00-b2b7-5c12fd496164 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f142988a4dd14a5f96eacea45c046f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:49:33.174126 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729e332d-3623-4b00-b2b7-5c12fd496164 service nova] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:33.174403 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729e332d-3623-4b00-b2b7-5c12fd496164 service nova] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:33.174672 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729e332d-3623-4b00-b2b7-5c12fd496164 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:33.176308 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-729e332d-3623-4b00-b2b7-5c12fd496164 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:33.272629 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-729e332d-3623-4b00-b2b7-5c12fd496164 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:49:33.274293 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-729e332d-3623-4b00-b2b7-5c12fd496164 service nova] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:49:33.294348 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 721/1434] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:49:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 386 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:49:34.369559 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf674d02-081b-4073-9262-b3059fd95494 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:34.370095 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf674d02-081b-4073-9262-b3059fd95494 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:34.370095 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf674d02-081b-4073-9262-b3059fd95494 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:34.371345 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 714/1435] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 25 00:49:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:49:34.377981 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846c58c4-63be-41a2-b32a-17d33852e4a0 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:34.378491 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846c58c4-63be-41a2-b32a-17d33852e4a0 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:34.378605 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846c58c4-63be-41a2-b32a-17d33852e4a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:34.457788 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-846c58c4-63be-41a2-b32a-17d33852e4a0 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:49:34.459549 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-846c58c4-63be-41a2-b32a-17d33852e4a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:34.502076 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:49:34.561851 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 722/1436] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 25 00:49:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:49:34.715689 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf674d02-081b-4073-9262-b3059fd95494 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:49:34.717134 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-bf674d02-081b-4073-9262-b3059fd95494 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:34.889705 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf674d02-081b-4073-9262-b3059fd95494 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:49:34.953057 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:49:35.020052 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-097be8a3-f765-48bd-a900-485bb0753c6e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f142988a4dd14a5f96eacea45c046f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:49:35.020702 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097be8a3-f765-48bd-a900-485bb0753c6e service nova] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:35.021181 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097be8a3-f765-48bd-a900-485bb0753c6e service nova] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:35.021587 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097be8a3-f765-48bd-a900-485bb0753c6e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:35.023271 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-097be8a3-f765-48bd-a900-485bb0753c6e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:35.118339 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-097be8a3-f765-48bd-a900-485bb0753c6e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:49:35.119917 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-097be8a3-f765-48bd-a900-485bb0753c6e service nova] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:49:35.138355 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 715/1437] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:49:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 423 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:49:36.219877 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec524f6-d611-4155-96dd-9b12bd054527 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:36.220480 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec524f6-d611-4155-96dd-9b12bd054527 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:36.220480 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec524f6-d611-4155-96dd-9b12bd054527 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:36.221768 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 723/1438] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 25 00:49:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:49:36.228912 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d880a0-4b36-4827-ac6c-68f5329935fe None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:36.229439 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d880a0-4b36-4827-ac6c-68f5329935fe None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:36.229533 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d880a0-4b36-4827-ac6c-68f5329935fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:36.306653 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-a5d880a0-4b36-4827-ac6c-68f5329935fe None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:49:36.307762 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-a5d880a0-4b36-4827-ac6c-68f5329935fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:36.309907 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:49:36.407314 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 716/1439] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 25 00:49:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:49:36.565143 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec524f6-d611-4155-96dd-9b12bd054527 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:49:36.566602 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-5ec524f6-d611-4155-96dd-9b12bd054527 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:36.714153 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec524f6-d611-4155-96dd-9b12bd054527 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:49:36.742555 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:49:36.851227 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00af3330-052c-4ace-b28f-c0aae263675e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f142988a4dd14a5f96eacea45c046f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:49:36.851735 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00af3330-052c-4ace-b28f-c0aae263675e service nova] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:36.852007 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00af3330-052c-4ace-b28f-c0aae263675e service nova] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:36.852267 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00af3330-052c-4ace-b28f-c0aae263675e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:36.853958 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-00af3330-052c-4ace-b28f-c0aae263675e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:36.946407 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00af3330-052c-4ace-b28f-c0aae263675e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:49:36.948007 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00af3330-052c-4ace-b28f-c0aae263675e service nova] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:49:36.966863 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 724/1440] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:49:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 402 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:49:39.116084 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff51af9-1237-4b62-9e04-947a0cbe88dc None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:39.116643 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff51af9-1237-4b62-9e04-947a0cbe88dc None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:39.116643 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff51af9-1237-4b62-9e04-947a0cbe88dc None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:39.117992 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 717/1441] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 25 00:49:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:49:39.124574 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1082a761-222c-44dd-9f8d-1ee812e9f11a None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:39.125066 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1082a761-222c-44dd-9f8d-1ee812e9f11a None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:39.125173 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1082a761-222c-44dd-9f8d-1ee812e9f11a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:39.171871 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:49:39.203012 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-1082a761-222c-44dd-9f8d-1ee812e9f11a None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:49:39.204231 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-1082a761-222c-44dd-9f8d-1ee812e9f11a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:39.308657 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 725/1442] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 25 00:49:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:49:39.600036 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff51af9-1237-4b62-9e04-947a0cbe88dc None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:49:39.601614 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-6ff51af9-1237-4b62-9e04-947a0cbe88dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:39.732635 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff51af9-1237-4b62-9e04-947a0cbe88dc None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:49:39.769619 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:49:39.818641 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a559902-761c-4a2c-80a3-df9a24c4177d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:49:39.819253 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a559902-761c-4a2c-80a3-df9a24c4177d service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:39.819523 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a559902-761c-4a2c-80a3-df9a24c4177d service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:39.819786 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a559902-761c-4a2c-80a3-df9a24c4177d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:39.821450 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-7a559902-761c-4a2c-80a3-df9a24c4177d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:39.945444 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a559902-761c-4a2c-80a3-df9a24c4177d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:49:39.946881 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a559902-761c-4a2c-80a3-df9a24c4177d service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:49:39.966533 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 718/1443] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:49:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:49:42.342305 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c010d0e5-cddb-4cea-9ce5-4c399159dc7a None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:42.342824 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c010d0e5-cddb-4cea-9ce5-4c399159dc7a None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:42.342824 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c010d0e5-cddb-4cea-9ce5-4c399159dc7a None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:42.344098 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 726/1444] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 25 00:49:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:49:42.350114 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b127ac31-854a-4ebb-90c7-478f1c17a2c6 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:42.350534 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b127ac31-854a-4ebb-90c7-478f1c17a2c6 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:42.350672 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b127ac31-854a-4ebb-90c7-478f1c17a2c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:42.428145 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-b127ac31-854a-4ebb-90c7-478f1c17a2c6 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:49:42.429419 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-b127ac31-854a-4ebb-90c7-478f1c17a2c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:42.442847 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:49:42.545561 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 719/1445] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 25 00:49:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:49:42.834163 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c010d0e5-cddb-4cea-9ce5-4c399159dc7a None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:49:42.835727 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-c010d0e5-cddb-4cea-9ce5-4c399159dc7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:42.948092 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c010d0e5-cddb-4cea-9ce5-4c399159dc7a None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:49:42.963617 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:49:43.028196 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f45bf6-df1b-48cc-b484-9f411f761cf2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:49:43.028805 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f45bf6-df1b-48cc-b484-9f411f761cf2 service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:43.029081 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f45bf6-df1b-48cc-b484-9f411f761cf2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:43.029444 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f45bf6-df1b-48cc-b484-9f411f761cf2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:43.031061 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-67f45bf6-df1b-48cc-b484-9f411f761cf2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:43.158032 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67f45bf6-df1b-48cc-b484-9f411f761cf2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:49:43.159377 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67f45bf6-df1b-48cc-b484-9f411f761cf2 service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:49:43.179721 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 727/1446] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:49:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 346 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:49:45.716160 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e6727f-db96-4588-a104-d1a634ea5afe None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:45.716732 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e6727f-db96-4588-a104-d1a634ea5afe None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:45.716732 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e6727f-db96-4588-a104-d1a634ea5afe None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:45.718115 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 720/1447] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 25 00:49:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:49:45.723354 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a805e579-0951-49da-8509-83cd94e4883b None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:45.723867 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a805e579-0951-49da-8509-83cd94e4883b None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:45.723965 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a805e579-0951-49da-8509-83cd94e4883b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:45.775404 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:49:45.794208 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-a805e579-0951-49da-8509-83cd94e4883b None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:49:45.795421 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-a805e579-0951-49da-8509-83cd94e4883b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:46.024799 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 728/1448] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 25 00:49:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 302 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:49:46.315376 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e6727f-db96-4588-a104-d1a634ea5afe None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:49:46.316937 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-c6e6727f-db96-4588-a104-d1a634ea5afe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:46.438801 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e6727f-db96-4588-a104-d1a634ea5afe None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:49:46.488634 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:49:46.530766 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb814a0-8dad-467a-b324-a5756e2848fd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:49:46.531365 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb814a0-8dad-467a-b324-a5756e2848fd service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:46.531636 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb814a0-8dad-467a-b324-a5756e2848fd service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:46.531897 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb814a0-8dad-467a-b324-a5756e2848fd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:46.533590 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-1eb814a0-8dad-467a-b324-a5756e2848fd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:46.658179 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb814a0-8dad-467a-b324-a5756e2848fd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:49:46.659525 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb814a0-8dad-467a-b324-a5756e2848fd service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:49:46.679254 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 721/1449] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:49:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 364 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:49:49.116565 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9feeaedc-0e6a-49e0-8e91-3c4fb17645b2 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:49.117121 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9feeaedc-0e6a-49e0-8e91-3c4fb17645b2 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:49.117121 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9feeaedc-0e6a-49e0-8e91-3c4fb17645b2 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:49.118545 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 729/1450] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 25 00:49:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:49:49.125577 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8361be4b-bbe0-4e0d-844b-490b3b75755f None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:49.126105 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8361be4b-bbe0-4e0d-844b-490b3b75755f None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:49.126105 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8361be4b-bbe0-4e0d-844b-490b3b75755f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:49.193286 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-8361be4b-bbe0-4e0d-844b-490b3b75755f None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:49:49.194594 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-8361be4b-bbe0-4e0d-844b-490b3b75755f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:49.214932 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:49:49.293180 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 722/1451] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 25 00:49:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:49:49.595611 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9feeaedc-0e6a-49e0-8e91-3c4fb17645b2 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:49:49.597138 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-9feeaedc-0e6a-49e0-8e91-3c4fb17645b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:49.708568 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9feeaedc-0e6a-49e0-8e91-3c4fb17645b2 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:49:49.729737 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:49:49.784634 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ef5d95-ac2e-49b3-a328-583bbfbfe1c6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:49:49.785144 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ef5d95-ac2e-49b3-a328-583bbfbfe1c6 service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:49.785449 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ef5d95-ac2e-49b3-a328-583bbfbfe1c6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:49.785718 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ef5d95-ac2e-49b3-a328-583bbfbfe1c6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:49.787544 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-f7ef5d95-ac2e-49b3-a328-583bbfbfe1c6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:49.899556 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7ef5d95-ac2e-49b3-a328-583bbfbfe1c6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:49:49.900978 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7ef5d95-ac2e-49b3-a328-583bbfbfe1c6 service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:49:49.918569 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 730/1452] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:49:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:49:52.167854 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85847ec4-c8a9-4949-8225-ebd4009e8762 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:52.168427 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85847ec4-c8a9-4949-8225-ebd4009e8762 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:52.168427 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85847ec4-c8a9-4949-8225-ebd4009e8762 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:52.169801 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 723/1453] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 25 00:49:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:49:52.175572 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e92e261-32e9-4d52-8f80-e35df7ca6732 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:52.176043 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e92e261-32e9-4d52-8f80-e35df7ca6732 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:52.176169 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e92e261-32e9-4d52-8f80-e35df7ca6732 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:52.244735 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:49:52.247790 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-2e92e261-32e9-4d52-8f80-e35df7ca6732 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:49:52.249028 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-2e92e261-32e9-4d52-8f80-e35df7ca6732 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:52.357619 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 731/1454] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 25 00:49:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:49:52.647143 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85847ec4-c8a9-4949-8225-ebd4009e8762 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:49:52.648671 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-85847ec4-c8a9-4949-8225-ebd4009e8762 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:52.756324 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85847ec4-c8a9-4949-8225-ebd4009e8762 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:49:52.808803 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:49:52.839799 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-307ed437-0afe-4afd-86f7-cf052202140a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:49:52.840403 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307ed437-0afe-4afd-86f7-cf052202140a service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:52.840672 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307ed437-0afe-4afd-86f7-cf052202140a service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:52.840930 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307ed437-0afe-4afd-86f7-cf052202140a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:52.842562 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-307ed437-0afe-4afd-86f7-cf052202140a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:52.963314 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-307ed437-0afe-4afd-86f7-cf052202140a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:49:52.965968 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-307ed437-0afe-4afd-86f7-cf052202140a service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:49:52.988097 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 724/1455] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:49:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 341 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:49:55.264129 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4557ede-a104-40f9-99b3-c4f1f2bdc2f7 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:55.264742 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4557ede-a104-40f9-99b3-c4f1f2bdc2f7 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:55.264742 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4557ede-a104-40f9-99b3-c4f1f2bdc2f7 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:55.266063 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 732/1456] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 25 00:49:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 00:49:55.271914 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f5bc3a-d918-4534-bf6f-628c30e6e92b None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:55.272354 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f5bc3a-d918-4534-bf6f-628c30e6e92b None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:55.272473 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f5bc3a-d918-4534-bf6f-628c30e6e92b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:55.339487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-64f5bc3a-d918-4534-bf6f-628c30e6e92b None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:49:55.340679 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-64f5bc3a-d918-4534-bf6f-628c30e6e92b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:55.377814 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:49:55.450463 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 725/1457] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 25 00:49:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 00:49:55.736015 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4557ede-a104-40f9-99b3-c4f1f2bdc2f7 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:49:55.737851 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-d4557ede-a104-40f9-99b3-c4f1f2bdc2f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:55.846739 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4557ede-a104-40f9-99b3-c4f1f2bdc2f7 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:49:55.876573 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:49:55.923883 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c525c547-a16a-4d25-a9db-6d686906a438 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:49:55.924483 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c525c547-a16a-4d25-a9db-6d686906a438 service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:55.924758 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c525c547-a16a-4d25-a9db-6d686906a438 service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:55.925022 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c525c547-a16a-4d25-a9db-6d686906a438 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:55.926805 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-c525c547-a16a-4d25-a9db-6d686906a438 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:56.039337 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c525c547-a16a-4d25-a9db-6d686906a438 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:49:56.040669 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c525c547-a16a-4d25-a9db-6d686906a438 service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:49:56.059441 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 733/1458] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:49:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:49:58.343667 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab44cfa-89ab-417c-928b-d442ba5a1ea4 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:58.344523 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab44cfa-89ab-417c-928b-d442ba5a1ea4 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:58.344523 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab44cfa-89ab-417c-928b-d442ba5a1ea4 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:58.346059 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 726/1459] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 25 00:49:58 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:49:58.352505 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4f7577-3f74-42a8-81eb-12098cf8e8c0 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:58.353008 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4f7577-3f74-42a8-81eb-12098cf8e8c0 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:58.353111 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4f7577-3f74-42a8-81eb-12098cf8e8c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:58.433776 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-9f4f7577-3f74-42a8-81eb-12098cf8e8c0 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:49:58.434984 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-9f4f7577-3f74-42a8-81eb-12098cf8e8c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:58.445736 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:49:58.546914 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 734/1460] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 25 00:49:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:49:58.838100 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab44cfa-89ab-417c-928b-d442ba5a1ea4 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:49:58.839728 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-4ab44cfa-89ab-417c-928b-d442ba5a1ea4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:58.980644 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab44cfa-89ab-417c-928b-d442ba5a1ea4 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:49:59.053938 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:49:59.075948 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-078106aa-c7da-46b1-abf3-fea69ca2f6f8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:49:59.076554 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078106aa-c7da-46b1-abf3-fea69ca2f6f8 service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:49:59.076824 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078106aa-c7da-46b1-abf3-fea69ca2f6f8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:49:59.077093 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078106aa-c7da-46b1-abf3-fea69ca2f6f8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:49:59.078760 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-078106aa-c7da-46b1-abf3-fea69ca2f6f8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:49:59.195185 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-078106aa-c7da-46b1-abf3-fea69ca2f6f8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:49:59.196528 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-078106aa-c7da-46b1-abf3-fea69ca2f6f8 service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:49:59.216619 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 727/1461] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:49:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:50:01.534212 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baaf0dc9-149e-45bb-9e1e-fc2454643ff8 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:50:01.534804 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baaf0dc9-149e-45bb-9e1e-fc2454643ff8 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:50:01.534804 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baaf0dc9-149e-45bb-9e1e-fc2454643ff8 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:50:01.536139 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 735/1462] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 25 00:50:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:50:01.541665 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076e59c8-e9f8-4554-afa5-d0c589f2dfd6 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:50:01.542146 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076e59c8-e9f8-4554-afa5-d0c589f2dfd6 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:50:01.542258 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076e59c8-e9f8-4554-afa5-d0c589f2dfd6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:50:01.609144 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-076e59c8-e9f8-4554-afa5-d0c589f2dfd6 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:50:01.610441 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-076e59c8-e9f8-4554-afa5-d0c589f2dfd6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:50:01.655419 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:50:01.716697 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 728/1463] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 25 00:50:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:50:02.002463 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baaf0dc9-149e-45bb-9e1e-fc2454643ff8 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:50:02.003997 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-baaf0dc9-149e-45bb-9e1e-fc2454643ff8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:50:02.117111 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baaf0dc9-149e-45bb-9e1e-fc2454643ff8 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:50:02.154809 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:50:02.192427 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-807f61b5-7816-4a2d-800b-850574be1bbf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:50:02.192964 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807f61b5-7816-4a2d-800b-850574be1bbf service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:50:02.193233 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807f61b5-7816-4a2d-800b-850574be1bbf service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:50:02.193524 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807f61b5-7816-4a2d-800b-850574be1bbf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:50:02.195138 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-807f61b5-7816-4a2d-800b-850574be1bbf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:50:02.304653 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-807f61b5-7816-4a2d-800b-850574be1bbf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:50:02.306015 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-807f61b5-7816-4a2d-800b-850574be1bbf service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:50:02.324555 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 736/1464] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:50:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:50:04.548793 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64948984-faac-481c-bc25-16b1ccd5456b None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:50:04.549456 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64948984-faac-481c-bc25-16b1ccd5456b None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:50:04.549456 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64948984-faac-481c-bc25-16b1ccd5456b None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:50:04.550688 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 729/1465] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 25 00:50:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:50:04.556438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b960cc9a-eeb3-4902-b8f5-e1eea428eee8 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:50:04.556902 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b960cc9a-eeb3-4902-b8f5-e1eea428eee8 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:50:04.557028 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b960cc9a-eeb3-4902-b8f5-e1eea428eee8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:50:04.623912 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-b960cc9a-eeb3-4902-b8f5-e1eea428eee8 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:50:04.625118 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-b960cc9a-eeb3-4902-b8f5-e1eea428eee8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:50:04.643523 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:50:04.724895 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 737/1466] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 25 00:50:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:50:05.015215 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64948984-faac-481c-bc25-16b1ccd5456b None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:50:05.016759 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-64948984-faac-481c-bc25-16b1ccd5456b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:50:05.123776 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64948984-faac-481c-bc25-16b1ccd5456b None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:50:05.193306 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:50:05.200357 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c91dc74-3299-4655-9db3-7339db99a2bc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:50:05.200959 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c91dc74-3299-4655-9db3-7339db99a2bc service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:50:05.201234 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c91dc74-3299-4655-9db3-7339db99a2bc service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:50:05.201536 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c91dc74-3299-4655-9db3-7339db99a2bc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:50:05.203136 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-4c91dc74-3299-4655-9db3-7339db99a2bc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:50:05.314105 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c91dc74-3299-4655-9db3-7339db99a2bc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:50:05.315422 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c91dc74-3299-4655-9db3-7339db99a2bc service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:50:05.334467 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 730/1467] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:50:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:50:07.581102 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb4f1a37-9f74-4766-a9d6-95083d621f84 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:50:07.581673 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb4f1a37-9f74-4766-a9d6-95083d621f84 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:50:07.581895 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb4f1a37-9f74-4766-a9d6-95083d621f84 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:50:07.583847 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 738/1468] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 25 00:50:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:50:07.589773 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5898286-c7d9-4f0a-bd75-f5f6779295cc None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:50:07.590277 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5898286-c7d9-4f0a-bd75-f5f6779295cc None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:50:07.590277 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5898286-c7d9-4f0a-bd75-f5f6779295cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:50:07.667666 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-b5898286-c7d9-4f0a-bd75-f5f6779295cc None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:50:07.668881 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-b5898286-c7d9-4f0a-bd75-f5f6779295cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:50:07.726662 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:50:07.784749 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 731/1469] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 25 00:50:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:50:08.073925 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb4f1a37-9f74-4766-a9d6-95083d621f84 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:50:08.075485 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-eb4f1a37-9f74-4766-a9d6-95083d621f84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:50:08.192727 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb4f1a37-9f74-4766-a9d6-95083d621f84 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:50:08.242160 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:50:08.269539 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02adb6e8-3c1c-4517-82a3-6c7d3e7d9694 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:50:08.270140 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02adb6e8-3c1c-4517-82a3-6c7d3e7d9694 service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:50:08.270524 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02adb6e8-3c1c-4517-82a3-6c7d3e7d9694 service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:50:08.271026 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02adb6e8-3c1c-4517-82a3-6c7d3e7d9694 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:50:08.272663 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-02adb6e8-3c1c-4517-82a3-6c7d3e7d9694 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:50:08.395156 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02adb6e8-3c1c-4517-82a3-6c7d3e7d9694 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:50:08.396491 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02adb6e8-3c1c-4517-82a3-6c7d3e7d9694 service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:50:08.415777 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 739/1470] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:50:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 342 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:50:17.784387 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=22 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:50:18.413430 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=66 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:50:29.271754 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e60d64-0ff2-4b7b-b9ff-02bf1578c50f None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:50:29.272420 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e60d64-0ff2-4b7b-b9ff-02bf1578c50f None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:50:29.272420 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e60d64-0ff2-4b7b-b9ff-02bf1578c50f None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:50:29.273687 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 732/1471] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:50:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:50:29.279239 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10432f3-bb97-4e49-970a-09c3f343d3fd None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:50:29.279694 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10432f3-bb97-4e49-970a-09c3f343d3fd None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:50:29.279803 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10432f3-bb97-4e49-970a-09c3f343d3fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:50:29.346804 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-c10432f3-bb97-4e49-970a-09c3f343d3fd None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:50:29.348697 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-c10432f3-bb97-4e49-970a-09c3f343d3fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:50:29.447941 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 740/1472] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:50:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:50:29.756521 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde7f608-3059-4f2f-8722-e546dbeeff25 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:50:29.756795 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde7f608-3059-4f2f-8722-e546dbeeff25 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:50:29.757060 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde7f608-3059-4f2f-8722-e546dbeeff25 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:50:29.758447 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 733/1473] 10.222.0.52 () {54 vars in 962 bytes} [Sun Sep 25 00:50:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:50:29.763482 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31637581-5bab-4060-b627-d2509449dd81 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:50:29.763759 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31637581-5bab-4060-b627-d2509449dd81 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:50:29.764033 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31637581-5bab-4060-b627-d2509449dd81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:50:29.851960 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-31637581-5bab-4060-b627-d2509449dd81 None None] MFA Rules not processed for user `4ebfc0abf1b842459a44ca9324252688`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:50:29.853180 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-31637581-5bab-4060-b627-d2509449dd81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:50:29.873983 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:50:29.933499 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 741/1474] 10.222.0.52 () {58 vars in 1059 bytes} [Sun Sep 25 00:50:29 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:50:29.941219 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde7f608-3059-4f2f-8722-e546dbeeff25 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:50:29.943290 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-bde7f608-3059-4f2f-8722-e546dbeeff25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:50:30.050145 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde7f608-3059-4f2f-8722-e546dbeeff25 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:50:30.128132 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8fcc151-ac70-4b29-9ca7-b2ea8012adc0 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4ebfc0abf1b842459a44ca9324252688', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:50:30.128728 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8fcc151-ac70-4b29-9ca7-b2ea8012adc0 service glance] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:50:30.129563 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8fcc151-ac70-4b29-9ca7-b2ea8012adc0 service glance] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:50:30.129829 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8fcc151-ac70-4b29-9ca7-b2ea8012adc0 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:50:30.131418 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-c8fcc151-ac70-4b29-9ca7-b2ea8012adc0 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:50:30.187867 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:50:30.250003 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8fcc151-ac70-4b29-9ca7-b2ea8012adc0 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:50:30.251439 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8fcc151-ac70-4b29-9ca7-b2ea8012adc0 service glance] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:50:30.271007 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 734/1475] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:50:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 330 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 00:50:32.547395 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1772097-e3b5-4ff8-9ecd-b882d292e415 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:50:32.547972 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1772097-e3b5-4ff8-9ecd-b882d292e415 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:50:32.547972 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1772097-e3b5-4ff8-9ecd-b882d292e415 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:50:32.549248 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 742/1476] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:50:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:50:32.555633 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c936146-bed7-47de-944f-345da431e559 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:50:32.556206 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c936146-bed7-47de-944f-345da431e559 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:50:32.556206 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c936146-bed7-47de-944f-345da431e559 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:50:32.624686 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-6c936146-bed7-47de-944f-345da431e559 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:50:32.625977 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-6c936146-bed7-47de-944f-345da431e559 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:50:32.728402 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 735/1477] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:50:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:50:33.005867 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1772097-e3b5-4ff8-9ecd-b882d292e415 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:50:33.007413 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-d1772097-e3b5-4ff8-9ecd-b882d292e415 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:50:33.134673 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1772097-e3b5-4ff8-9ecd-b882d292e415 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:50:33.184120 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:50:33.229674 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecacb536-fd39-462d-a572-8545c1d28db8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:50:33.230271 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecacb536-fd39-462d-a572-8545c1d28db8 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:50:33.230542 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecacb536-fd39-462d-a572-8545c1d28db8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:50:33.230806 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecacb536-fd39-462d-a572-8545c1d28db8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:50:33.231439 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecacb536-fd39-462d-a572-8545c1d28db8 admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:50:33.233159 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: WARNING py.warnings [None req-ecacb536-fd39-462d-a572-8545c1d28db8 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 25 00:50:33.233159 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: warnings.warn(msg) Sep 25 00:50:33.233159 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: Sep 25 00:50:33.234323 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecacb536-fd39-462d-a572-8545c1d28db8 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:50:33.253099 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 743/1478] 10.222.0.52 () {56 vars in 1108 bytes} [Sun Sep 25 00:50:33 2022] GET /identity/v3/projects => generated 2553 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:50:35.477473 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38743bc4-b195-41d6-8760-7ca0a6a45d2d None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:50:35.478155 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38743bc4-b195-41d6-8760-7ca0a6a45d2d None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:50:35.478155 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38743bc4-b195-41d6-8760-7ca0a6a45d2d None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:50:35.479489 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 736/1479] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:50:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:50:35.485767 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f51be4d-af02-4f74-959c-0714c00764e2 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:50:35.486472 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f51be4d-af02-4f74-959c-0714c00764e2 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:50:35.486472 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f51be4d-af02-4f74-959c-0714c00764e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:50:35.565942 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-3f51be4d-af02-4f74-959c-0714c00764e2 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:50:35.567602 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-3f51be4d-af02-4f74-959c-0714c00764e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:50:35.678260 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 744/1480] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:50:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:50:35.979628 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38743bc4-b195-41d6-8760-7ca0a6a45d2d None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:50:35.981177 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-38743bc4-b195-41d6-8760-7ca0a6a45d2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:50:35.991488 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:50:36.140952 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38743bc4-b195-41d6-8760-7ca0a6a45d2d None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:50:36.228460 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:50:36.263358 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe436b4-aa94-44ff-9987-7350702a27c9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f142988a4dd14a5f96eacea45c046f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:50:36.263977 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe436b4-aa94-44ff-9987-7350702a27c9 service nova] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:50:36.264252 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe436b4-aa94-44ff-9987-7350702a27c9 service nova] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:50:36.264518 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe436b4-aa94-44ff-9987-7350702a27c9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:50:36.266192 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-9fe436b4-aa94-44ff-9987-7350702a27c9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:50:36.369097 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fe436b4-aa94-44ff-9987-7350702a27c9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:50:36.370724 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fe436b4-aa94-44ff-9987-7350702a27c9 service nova] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:50:36.389319 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 737/1481] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:50:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 410 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:50:38.805248 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39354093-241b-4acc-aef9-b8200947c592 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:50:38.805889 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39354093-241b-4acc-aef9-b8200947c592 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:50:38.805889 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39354093-241b-4acc-aef9-b8200947c592 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:50:38.807061 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 745/1482] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:50:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:50:38.812443 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74663fbd-d9a9-4fea-bf7b-e9ec22b44fa1 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:50:38.812920 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74663fbd-d9a9-4fea-bf7b-e9ec22b44fa1 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:50:38.813135 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74663fbd-d9a9-4fea-bf7b-e9ec22b44fa1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:50:38.883948 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-74663fbd-d9a9-4fea-bf7b-e9ec22b44fa1 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:50:38.885172 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-74663fbd-d9a9-4fea-bf7b-e9ec22b44fa1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:50:38.935000 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:50:38.991686 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 738/1483] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:50:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:50:39.282599 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39354093-241b-4acc-aef9-b8200947c592 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:50:39.284178 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-39354093-241b-4acc-aef9-b8200947c592 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:50:39.308270 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:50:39.398484 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39354093-241b-4acc-aef9-b8200947c592 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:50:39.475160 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a2c3c0-f388-4a2f-9e71-618fa040e012 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:50:39.475771 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a2c3c0-f388-4a2f-9e71-618fa040e012 service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:50:39.476046 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a2c3c0-f388-4a2f-9e71-618fa040e012 service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:50:39.476309 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a2c3c0-f388-4a2f-9e71-618fa040e012 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:50:39.477986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-87a2c3c0-f388-4a2f-9e71-618fa040e012 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:50:39.557654 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:50:39.594096 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a2c3c0-f388-4a2f-9e71-618fa040e012 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:50:39.595505 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a2c3c0-f388-4a2f-9e71-618fa040e012 service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:50:39.614614 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 746/1484] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:50:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 333 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:50:41.898071 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89fa4b5-9461-4314-b86a-e7f1d49ff469 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:50:41.898593 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89fa4b5-9461-4314-b86a-e7f1d49ff469 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:50:41.898593 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89fa4b5-9461-4314-b86a-e7f1d49ff469 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:50:41.899877 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 739/1485] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:50:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:50:41.905471 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a231c3f8-bbb7-418c-bfa9-32da5d9daef0 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:50:41.905946 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a231c3f8-bbb7-418c-bfa9-32da5d9daef0 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:50:41.906053 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a231c3f8-bbb7-418c-bfa9-32da5d9daef0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:50:41.978089 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-a231c3f8-bbb7-418c-bfa9-32da5d9daef0 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:50:41.979310 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-a231c3f8-bbb7-418c-bfa9-32da5d9daef0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:50:42.080788 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 747/1486] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:50:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:50:42.374824 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f89fa4b5-9461-4314-b86a-e7f1d49ff469 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:50:42.376379 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-f89fa4b5-9461-4314-b86a-e7f1d49ff469 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:50:42.484844 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f89fa4b5-9461-4314-b86a-e7f1d49ff469 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:50:42.571242 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:50:42.573039 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c03cf5d-5164-453a-b7dd-408a19e89bf4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:50:42.573756 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c03cf5d-5164-453a-b7dd-408a19e89bf4 service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:50:42.574290 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c03cf5d-5164-453a-b7dd-408a19e89bf4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:50:42.574637 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c03cf5d-5164-453a-b7dd-408a19e89bf4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:50:42.576634 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-9c03cf5d-5164-453a-b7dd-408a19e89bf4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:50:42.694103 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c03cf5d-5164-453a-b7dd-408a19e89bf4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:50:42.695473 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c03cf5d-5164-453a-b7dd-408a19e89bf4 service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:50:42.834584 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 740/1487] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:50:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 460 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:50:45.173343 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f56c30-8d2a-448e-a73e-162896c8a457 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:50:45.174035 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f56c30-8d2a-448e-a73e-162896c8a457 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:50:45.174035 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f56c30-8d2a-448e-a73e-162896c8a457 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:50:45.175264 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 748/1488] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:50:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:50:45.181195 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee98f840-49c0-4d3d-ab9b-88958d472725 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:50:45.181758 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee98f840-49c0-4d3d-ab9b-88958d472725 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:50:45.181758 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee98f840-49c0-4d3d-ab9b-88958d472725 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:50:45.255955 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-ee98f840-49c0-4d3d-ab9b-88958d472725 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:50:45.257176 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-ee98f840-49c0-4d3d-ab9b-88958d472725 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:50:45.317095 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:50:45.363568 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 741/1489] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:50:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:50:45.659364 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f56c30-8d2a-448e-a73e-162896c8a457 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:50:45.661557 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-f2f56c30-8d2a-448e-a73e-162896c8a457 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:50:45.700138 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:50:45.771778 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f56c30-8d2a-448e-a73e-162896c8a457 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:50:45.848407 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7247057-a468-4863-8901-4f7b97eb6a7a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:50:45.849145 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7247057-a468-4863-8901-4f7b97eb6a7a service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:50:45.849444 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7247057-a468-4863-8901-4f7b97eb6a7a service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:50:45.849709 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7247057-a468-4863-8901-4f7b97eb6a7a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:50:45.851306 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-b7247057-a468-4863-8901-4f7b97eb6a7a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:50:45.933381 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:50:45.967144 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7247057-a468-4863-8901-4f7b97eb6a7a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:50:45.968586 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7247057-a468-4863-8901-4f7b97eb6a7a service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:50:45.989406 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 749/1490] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:50:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 331 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:50:48.447557 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566250e6-bcb7-4789-b4c4-af1ba5505e88 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:50:48.448100 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566250e6-bcb7-4789-b4c4-af1ba5505e88 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:50:48.448100 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566250e6-bcb7-4789-b4c4-af1ba5505e88 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:50:48.449373 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 742/1491] 10.222.0.52 () {54 vars in 927 bytes} [Sun Sep 25 00:50:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:50:48.454775 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68ba8eb-50a0-484d-b3a9-b52392352669 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:50:48.455210 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68ba8eb-50a0-484d-b3a9-b52392352669 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:50:48.455439 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68ba8eb-50a0-484d-b3a9-b52392352669 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:50:48.526241 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-f68ba8eb-50a0-484d-b3a9-b52392352669 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:50:48.527445 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-f68ba8eb-50a0-484d-b3a9-b52392352669 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:50:48.626147 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 750/1492] 10.222.0.52 () {58 vars in 1024 bytes} [Sun Sep 25 00:50:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:50:48.923735 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-566250e6-bcb7-4789-b4c4-af1ba5505e88 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:50:48.925404 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-566250e6-bcb7-4789-b4c4-af1ba5505e88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:50:49.033785 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-566250e6-bcb7-4789-b4c4-af1ba5505e88 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:50:49.109173 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5e4705-7b48-4f26-84c3-2041963da131 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:50:49.109833 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5e4705-7b48-4f26-84c3-2041963da131 service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:50:49.110103 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5e4705-7b48-4f26-84c3-2041963da131 service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:50:49.110366 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5e4705-7b48-4f26-84c3-2041963da131 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:50:49.111957 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-fd5e4705-7b48-4f26-84c3-2041963da131 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:50:49.122180 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:50:49.224985 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd5e4705-7b48-4f26-84c3-2041963da131 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:50:49.226685 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd5e4705-7b48-4f26-84c3-2041963da131 service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:50:49.249200 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 743/1493] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:50:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 326 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:50:58.624641 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=82 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:50:59.246790 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=50 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:52:05.700513 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6cd3777-6729-4cfd-9d13-ba382eb16e72 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:52:05.701136 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6cd3777-6729-4cfd-9d13-ba382eb16e72 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:52:05.701136 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6cd3777-6729-4cfd-9d13-ba382eb16e72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:52:05.785934 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-b6cd3777-6729-4cfd-9d13-ba382eb16e72 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:52:05.787317 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-b6cd3777-6729-4cfd-9d13-ba382eb16e72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:52:05.907107 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 751/1494] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:52:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:52:05.916793 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b41536a-f641-4909-965b-40ef273e8305 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:52:05.917451 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b41536a-f641-4909-965b-40ef273e8305 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:52:05.917781 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b41536a-f641-4909-965b-40ef273e8305 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:52:06.007226 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-1b41536a-f641-4909-965b-40ef273e8305 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:52:06.008442 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-1b41536a-f641-4909-965b-40ef273e8305 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:52:06.115498 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 744/1495] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:52:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:52:06.120060 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6cd3777-6729-4cfd-9d13-ba382eb16e72 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:52:06.121273 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-b6cd3777-6729-4cfd-9d13-ba382eb16e72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:52:06.133933 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6cd3777-6729-4cfd-9d13-ba382eb16e72 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:52:06.200531 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:52:06.239427 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9caeafc-cea9-4231-b1f5-6d324a170fd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:52:06.240034 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9caeafc-cea9-4231-b1f5-6d324a170fd6 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:52:06.240298 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9caeafc-cea9-4231-b1f5-6d324a170fd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:52:06.240563 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9caeafc-cea9-4231-b1f5-6d324a170fd6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:52:06.241197 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9caeafc-cea9-4231-b1f5-6d324a170fd6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:52:06.241519 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9caeafc-cea9-4231-b1f5-6d324a170fd6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:52:06.242521 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: WARNING py.warnings [None req-d9caeafc-cea9-4231-b1f5-6d324a170fd6 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 25 00:52:06.242521 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: warnings.warn(msg) Sep 25 00:52:06.242521 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: Sep 25 00:52:06.243401 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9caeafc-cea9-4231-b1f5-6d324a170fd6 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:52:06.255806 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 752/1496] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 00:52:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:52:06.264390 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b41536a-f641-4909-965b-40ef273e8305 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:52:06.274677 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b41536a-f641-4909-965b-40ef273e8305 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:52:06.338785 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:52:06.377220 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d17e6e-d4e7-4b4f-a6c5-16d990e86ad0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:52:06.378382 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d17e6e-d4e7-4b4f-a6c5-16d990e86ad0 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:52:06.378705 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d17e6e-d4e7-4b4f-a6c5-16d990e86ad0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:52:06.379021 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d17e6e-d4e7-4b4f-a6c5-16d990e86ad0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:52:06.379713 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9d17e6e-d4e7-4b4f-a6c5-16d990e86ad0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:52:06.381553 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: WARNING py.warnings [None req-a9d17e6e-d4e7-4b4f-a6c5-16d990e86ad0 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 25 00:52:06.381553 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: warnings.warn(msg) Sep 25 00:52:06.381553 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: Sep 25 00:52:06.382742 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9d17e6e-d4e7-4b4f-a6c5-16d990e86ad0 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:52:06.447887 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 745/1497] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:52:06 2022] POST /identity/v3/projects => generated 377 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 00:52:06.452142 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9caeafc-cea9-4231-b1f5-6d324a170fd6 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:52:06.453673 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-d9caeafc-cea9-4231-b1f5-6d324a170fd6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:52:06.492709 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9caeafc-cea9-4231-b1f5-6d324a170fd6 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:52:06.601416 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e90172-478f-4cf2-914c-38455fa3cb96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:52:06.602016 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e90172-478f-4cf2-914c-38455fa3cb96 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:52:06.602286 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e90172-478f-4cf2-914c-38455fa3cb96 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:52:06.602549 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e90172-478f-4cf2-914c-38455fa3cb96 admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:52:06.603194 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4e90172-478f-4cf2-914c-38455fa3cb96 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:52:06.604215 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: WARNING py.warnings [None req-f4e90172-478f-4cf2-914c-38455fa3cb96 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 25 00:52:06.604215 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: warnings.warn(msg) Sep 25 00:52:06.604215 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: Sep 25 00:52:06.605458 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4e90172-478f-4cf2-914c-38455fa3cb96 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:52:06.668906 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 753/1498] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:52:06 2022] POST /identity/v3/users => generated 289 bytes in 217 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:52:06.673087 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d17e6e-d4e7-4b4f-a6c5-16d990e86ad0 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:52:06.675377 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-a9d17e6e-d4e7-4b4f-a6c5-16d990e86ad0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:52:06.741655 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d17e6e-d4e7-4b4f-a6c5-16d990e86ad0 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:52:06.832633 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:52:06.845670 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077615e1-61bd-4590-8e61-db3d228b1a99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:52:06.846343 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077615e1-61bd-4590-8e61-db3d228b1a99 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:52:06.846615 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077615e1-61bd-4590-8e61-db3d228b1a99 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:52:06.846878 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077615e1-61bd-4590-8e61-db3d228b1a99 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:52:06.847519 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-077615e1-61bd-4590-8e61-db3d228b1a99 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:52:06.848521 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: WARNING py.warnings [None req-077615e1-61bd-4590-8e61-db3d228b1a99 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 25 00:52:06.848521 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: warnings.warn(msg) Sep 25 00:52:06.848521 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: Sep 25 00:52:06.849566 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-077615e1-61bd-4590-8e61-db3d228b1a99 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:52:06.863667 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 746/1499] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:52:06 2022] GET /identity/v3/roles => generated 1420 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:52:06.867963 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e90172-478f-4cf2-914c-38455fa3cb96 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:52:06.869592 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-f4e90172-478f-4cf2-914c-38455fa3cb96 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:52:06.881986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e90172-478f-4cf2-914c-38455fa3cb96 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:52:06.908359 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:52:06.986804 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6db16f43-68be-43c5-be7f-81f60f2722d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:52:06.987604 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db16f43-68be-43c5-be7f-81f60f2722d4 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:52:06.987876 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db16f43-68be-43c5-be7f-81f60f2722d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:52:06.988135 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db16f43-68be-43c5-be7f-81f60f2722d4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:52:06.989025 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6db16f43-68be-43c5-be7f-81f60f2722d4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:52:06.990071 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: WARNING py.warnings [None req-6db16f43-68be-43c5-be7f-81f60f2722d4 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 25 00:52:06.990071 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: warnings.warn(msg) Sep 25 00:52:06.990071 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: Sep 25 00:52:06.990954 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6db16f43-68be-43c5-be7f-81f60f2722d4 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:52:07.004705 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 754/1500] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:52:06 2022] GET /identity/v3/roles => generated 1420 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:52:07.009607 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077615e1-61bd-4590-8e61-db3d228b1a99 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:52:07.011155 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-077615e1-61bd-4590-8e61-db3d228b1a99 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:52:07.047068 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077615e1-61bd-4590-8e61-db3d228b1a99 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:52:07.184394 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87cad724-e687-4c42-8f8e-3e350facee26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:52:07.185062 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87cad724-e687-4c42-8f8e-3e350facee26 admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:52:07.185377 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87cad724-e687-4c42-8f8e-3e350facee26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:52:07.185645 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87cad724-e687-4c42-8f8e-3e350facee26 admin admin] PATH_INFO: `/v3/projects/e185fae5d18d43bd846f04c42cfbee1e/users/5f2f801fb9444392bda03f4d7e1e8e67/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:52:07.212519 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87cad724-e687-4c42-8f8e-3e350facee26 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e185fae5d18d43bd846f04c42cfbee1e, user_id=5f2f801fb9444392bda03f4d7e1e8e67, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:52:07.213591 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: WARNING py.warnings [None req-87cad724-e687-4c42-8f8e-3e350facee26 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 25 00:52:07.213591 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: warnings.warn(msg) Sep 25 00:52:07.213591 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: Sep 25 00:52:07.215557 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87cad724-e687-4c42-8f8e-3e350facee26 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:52:07.250802 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 747/1501] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:52:07 2022] PUT /identity/v3/projects/e185fae5d18d43bd846f04c42cfbee1e/users/5f2f801fb9444392bda03f4d7e1e8e67/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 242 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:52:07.265232 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5b14de-20d7-4c10-9a35-205b5c6cd5db None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:52:07.265782 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5b14de-20d7-4c10-9a35-205b5c6cd5db None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:52:07.266252 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5b14de-20d7-4c10-9a35-205b5c6cd5db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:52:07.353180 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-3c5b14de-20d7-4c10-9a35-205b5c6cd5db None None] MFA Rules not processed for user `5f2f801fb9444392bda03f4d7e1e8e67`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:52:07.354570 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-3c5b14de-20d7-4c10-9a35-205b5c6cd5db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:52:07.447189 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 755/1502] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:52:07 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:52:07.462487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc47383-9dee-4f64-b175-0a0fa6f4127d None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:52:07.462766 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc47383-9dee-4f64-b175-0a0fa6f4127d None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:52:07.463036 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc47383-9dee-4f64-b175-0a0fa6f4127d None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:52:07.464425 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 748/1503] 10.222.0.52 () {54 vars in 963 bytes} [Sun Sep 25 00:52:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 00:52:07.469968 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66cfaf5d-22e5-430d-8e7a-9f7c1a421e89 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:52:07.470324 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66cfaf5d-22e5-430d-8e7a-9f7c1a421e89 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:52:07.470643 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66cfaf5d-22e5-430d-8e7a-9f7c1a421e89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:52:07.519454 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:52:07.593912 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-66cfaf5d-22e5-430d-8e7a-9f7c1a421e89 None None] MFA Rules not processed for user `944f10f38ee2428a85599f6a714ac806`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:52:07.595651 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-66cfaf5d-22e5-430d-8e7a-9f7c1a421e89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:52:07.692224 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 756/1504] 10.222.0.52 () {58 vars in 1060 bytes} [Sun Sep 25 00:52:07 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 223 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:52:07.701975 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc47383-9dee-4f64-b175-0a0fa6f4127d None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:52:07.703507 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-3cc47383-9dee-4f64-b175-0a0fa6f4127d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:52:07.755403 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:52:07.822078 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc47383-9dee-4f64-b175-0a0fa6f4127d None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:52:07.904509 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8731342-488d-4fe0-a026-f96745515fe9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:52:07.905107 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8731342-488d-4fe0-a026-f96745515fe9 service cinder] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:52:07.905415 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8731342-488d-4fe0-a026-f96745515fe9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:52:07.905684 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8731342-488d-4fe0-a026-f96745515fe9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:52:07.907276 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-c8731342-488d-4fe0-a026-f96745515fe9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:52:08.011158 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8731342-488d-4fe0-a026-f96745515fe9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:52:08.012592 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8731342-488d-4fe0-a026-f96745515fe9 service cinder] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:52:08.016187 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:52:08.033929 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 749/1505] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:52:07 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 332 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:52:08.085302 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ea6017-ee08-4bd3-98e7-8511edf6e854 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:52:08.085982 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ea6017-ee08-4bd3-98e7-8511edf6e854 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:52:08.085982 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ea6017-ee08-4bd3-98e7-8511edf6e854 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:52:08.087181 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 757/1506] 10.222.0.52 () {52 vars in 830 bytes} [Sun Sep 25 00:52:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:52:08.091125 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8731342-488d-4fe0-a026-f96745515fe9 service cinder] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:52:08.092728 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-c8731342-488d-4fe0-a026-f96745515fe9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:52:08.206452 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8731342-488d-4fe0-a026-f96745515fe9 service cinder] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:52:08.309345 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e8d1090-d9ff-4e9c-b55c-8696179fd612 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:52:08.309984 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8d1090-d9ff-4e9c-b55c-8696179fd612 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:52:08.310244 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8d1090-d9ff-4e9c-b55c-8696179fd612 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:52:08.310508 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8d1090-d9ff-4e9c-b55c-8696179fd612 admin admin] PATH_INFO: `/v3/users/5f2f801fb9444392bda03f4d7e1e8e67` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:52:08.323770 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e8d1090-d9ff-4e9c-b55c-8696179fd612 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5f2f801fb9444392bda03f4d7e1e8e67)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:52:08.324709 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: WARNING py.warnings [None req-7e8d1090-d9ff-4e9c-b55c-8696179fd612 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 25 00:52:08.324709 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: warnings.warn(msg) Sep 25 00:52:08.324709 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: Sep 25 00:52:08.325956 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e8d1090-d9ff-4e9c-b55c-8696179fd612 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:52:08.336721 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:52:08.399036 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-7e8d1090-d9ff-4e9c-b55c-8696179fd612 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5f2f801fb9444392bda03f4d7e1e8e67'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:52:08.399630 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-7e8d1090-d9ff-4e9c-b55c-8696179fd612 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5f2f801fb9444392bda03f4d7e1e8e67'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:52:08.411538 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-7e8d1090-d9ff-4e9c-b55c-8696179fd612 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5f2f801fb9444392bda03f4d7e1e8e67'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:52:08.411952 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-7e8d1090-d9ff-4e9c-b55c-8696179fd612 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5f2f801fb9444392bda03f4d7e1e8e67'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:52:08.431317 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-7e8d1090-d9ff-4e9c-b55c-8696179fd612 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5f2f801fb9444392bda03f4d7e1e8e67'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:52:08.468210 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 750/1507] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:52:08 2022] DELETE /identity/v3/users/5f2f801fb9444392bda03f4d7e1e8e67 => generated 0 bytes in 378 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:52:08.473734 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d211a913-7ed4-4f12-9a8b-1cefaccc3429 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:52:08.474061 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d211a913-7ed4-4f12-9a8b-1cefaccc3429 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:52:08.474372 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d211a913-7ed4-4f12-9a8b-1cefaccc3429 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:52:08.555234 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-d211a913-7ed4-4f12-9a8b-1cefaccc3429 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:52:08.556461 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-d211a913-7ed4-4f12-9a8b-1cefaccc3429 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:52:08.643364 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:52:08.657790 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 758/1508] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:52:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:52:08.669410 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e8d1090-d9ff-4e9c-b55c-8696179fd612 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:52:08.670955 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-7e8d1090-d9ff-4e9c-b55c-8696179fd612 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:52:08.789354 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e8d1090-d9ff-4e9c-b55c-8696179fd612 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:52:08.849472 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:52:08.872175 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-828a633f-0053-4d56-910a-4ff0dd6ab50e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:52:08.872805 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828a633f-0053-4d56-910a-4ff0dd6ab50e service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:52:08.873069 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828a633f-0053-4d56-910a-4ff0dd6ab50e service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:52:08.873374 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828a633f-0053-4d56-910a-4ff0dd6ab50e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:52:08.874985 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-828a633f-0053-4d56-910a-4ff0dd6ab50e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:52:08.993038 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-828a633f-0053-4d56-910a-4ff0dd6ab50e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:52:08.994555 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-828a633f-0053-4d56-910a-4ff0dd6ab50e service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:52:09.027763 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 751/1509] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:52:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 359 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:52:09.426569 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d211a913-7ed4-4f12-9a8b-1cefaccc3429 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:52:09.427828 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-d211a913-7ed4-4f12-9a8b-1cefaccc3429 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:52:09.440335 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d211a913-7ed4-4f12-9a8b-1cefaccc3429 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:52:09.550329 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3abbe5a6-2eb3-41c5-b644-18ff3f5d77c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:52:09.550873 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3abbe5a6-2eb3-41c5-b644-18ff3f5d77c2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:52:09.551142 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3abbe5a6-2eb3-41c5-b644-18ff3f5d77c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:52:09.551405 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3abbe5a6-2eb3-41c5-b644-18ff3f5d77c2 admin admin] PATH_INFO: `/v3/projects/e185fae5d18d43bd846f04c42cfbee1e` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:52:09.561892 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3abbe5a6-2eb3-41c5-b644-18ff3f5d77c2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e185fae5d18d43bd846f04c42cfbee1e)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:52:09.562933 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: WARNING py.warnings [None req-3abbe5a6-2eb3-41c5-b644-18ff3f5d77c2 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 25 00:52:09.562933 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: warnings.warn(msg) Sep 25 00:52:09.562933 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: Sep 25 00:52:09.565129 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3abbe5a6-2eb3-41c5-b644-18ff3f5d77c2 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:52:09.630825 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-3abbe5a6-2eb3-41c5-b644-18ff3f5d77c2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e185fae5d18d43bd846f04c42cfbee1e', 'request_id': 'req-3abbe5a6-2eb3-41c5-b644-18ff3f5d77c2'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:52:09.641492 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-3abbe5a6-2eb3-41c5-b644-18ff3f5d77c2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e185fae5d18d43bd846f04c42cfbee1e', 'request_id': 'req-3abbe5a6-2eb3-41c5-b644-18ff3f5d77c2'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:52:09.648558 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-3abbe5a6-2eb3-41c5-b644-18ff3f5d77c2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e185fae5d18d43bd846f04c42cfbee1e', 'request_id': 'req-3abbe5a6-2eb3-41c5-b644-18ff3f5d77c2'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:52:09.682125 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-3abbe5a6-2eb3-41c5-b644-18ff3f5d77c2 admin admin] The token cache is being invalidate because project e185fae5d18d43bd846f04c42cfbee1e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81386) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:52:09.682411 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-3abbe5a6-2eb3-41c5-b644-18ff3f5d77c2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:52:09.682941 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-3abbe5a6-2eb3-41c5-b644-18ff3f5d77c2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:52:09.684269 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 759/1510] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:52:09 2022] DELETE /identity/v3/projects/e185fae5d18d43bd846f04c42cfbee1e => generated 0 bytes in 258 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:52:19.024355 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=62 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:52:19.679940 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:53:43.545951 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d17707e-cc9c-435f-8654-89103476a935 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:53:43.546596 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d17707e-cc9c-435f-8654-89103476a935 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:53:43.546596 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d17707e-cc9c-435f-8654-89103476a935 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:53:43.547817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 752/1511] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 25 00:53:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:53:43.553138 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-647831d5-febd-4fa5-bc9f-1a082bcc6171 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:53:43.553634 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-647831d5-febd-4fa5-bc9f-1a082bcc6171 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:53:43.553739 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-647831d5-febd-4fa5-bc9f-1a082bcc6171 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:53:43.640775 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-647831d5-febd-4fa5-bc9f-1a082bcc6171 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:53:43.642037 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-647831d5-febd-4fa5-bc9f-1a082bcc6171 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:53:43.750075 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 760/1512] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 25 00:53:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:53:43.914671 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d17707e-cc9c-435f-8654-89103476a935 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:53:43.917400 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-1d17707e-cc9c-435f-8654-89103476a935 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:53:44.063586 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d17707e-cc9c-435f-8654-89103476a935 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:53:44.069041 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a02a98e-d5f3-4d7b-9a79-2db19c8f69ac service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f142988a4dd14a5f96eacea45c046f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:53:44.069664 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a02a98e-d5f3-4d7b-9a79-2db19c8f69ac service nova] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:53:44.069938 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a02a98e-d5f3-4d7b-9a79-2db19c8f69ac service nova] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:53:44.070198 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a02a98e-d5f3-4d7b-9a79-2db19c8f69ac service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:53:44.078590 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a02a98e-d5f3-4d7b-9a79-2db19c8f69ac service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:53:44.080153 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a02a98e-d5f3-4d7b-9a79-2db19c8f69ac service nova] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:53:44.089971 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 753/1513] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:53:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:53:44.994468 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a64ecbad-ca5f-474e-99a4-e1762d56386d None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:53:44.995056 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a64ecbad-ca5f-474e-99a4-e1762d56386d None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:53:44.995056 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a64ecbad-ca5f-474e-99a4-e1762d56386d None None] PATH_INFO: `/v3` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:53:44.996332 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 761/1514] 10.222.0.57 () {54 vars in 925 bytes} [Sun Sep 25 00:53:44 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:53:45.032405 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ab53d8-d6cb-447d-a0ee-3f6cfeb06892 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:53:45.032944 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ab53d8-d6cb-447d-a0ee-3f6cfeb06892 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:53:45.032944 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ab53d8-d6cb-447d-a0ee-3f6cfeb06892 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:53:45.096080 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-04ab53d8-d6cb-447d-a0ee-3f6cfeb06892 None None] MFA Rules not processed for user `11a80b55dcd444df901c8bf80f2a2c26`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:53:45.097389 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-04ab53d8-d6cb-447d-a0ee-3f6cfeb06892 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:53:45.170326 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:53:45.176228 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 754/1515] 10.222.0.57 () {58 vars in 1015 bytes} [Sun Sep 25 00:53:45 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:53:45.192881 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdff70a8-eef2-4f54-b24b-64855f200bb1 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:53:45.193160 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdff70a8-eef2-4f54-b24b-64855f200bb1 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:53:45.193477 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdff70a8-eef2-4f54-b24b-64855f200bb1 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:53:45.194827 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 762/1516] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 25 00:53:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:53:45.201526 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9742d30b-498f-4ce4-9bff-057dcafbedc0 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:53:45.201868 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9742d30b-498f-4ce4-9bff-057dcafbedc0 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:53:45.202138 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9742d30b-498f-4ce4-9bff-057dcafbedc0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:53:45.223189 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-9742d30b-498f-4ce4-9bff-057dcafbedc0 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:53:45.224381 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-9742d30b-498f-4ce4-9bff-057dcafbedc0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:53:45.232592 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 755/1517] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 25 00:53:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:53:45.387760 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdff70a8-eef2-4f54-b24b-64855f200bb1 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:53:45.402383 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdff70a8-eef2-4f54-b24b-64855f200bb1 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:53:45.408110 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e8d6009-b0d4-4c09-9ba8-907ad39edba4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f142988a4dd14a5f96eacea45c046f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:53:45.408703 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e8d6009-b0d4-4c09-9ba8-907ad39edba4 service nova] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:53:45.408977 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e8d6009-b0d4-4c09-9ba8-907ad39edba4 service nova] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:53:45.409244 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e8d6009-b0d4-4c09-9ba8-907ad39edba4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:53:45.417539 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e8d6009-b0d4-4c09-9ba8-907ad39edba4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:53:45.419118 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e8d6009-b0d4-4c09-9ba8-907ad39edba4 service nova] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:53:45.429115 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 763/1518] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:53:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:53:54.091867 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ff4575-fbcd-41ef-a159-6823a9f2e266 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:53:54.092422 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ff4575-fbcd-41ef-a159-6823a9f2e266 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:53:54.092422 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ff4575-fbcd-41ef-a159-6823a9f2e266 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:53:54.094538 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 756/1519] 10.222.0.52 () {54 vars in 961 bytes} [Sun Sep 25 00:53:54 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:53:54.100202 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f6e22f-91f3-4422-b865-ec211f6e5cee None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:53:54.100855 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f6e22f-91f3-4422-b865-ec211f6e5cee None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:53:54.100855 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f6e22f-91f3-4422-b865-ec211f6e5cee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:53:54.125322 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-f2f6e22f-91f3-4422-b865-ec211f6e5cee None None] MFA Rules not processed for user `11a80b55dcd444df901c8bf80f2a2c26`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:53:54.126557 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-f2f6e22f-91f3-4422-b865-ec211f6e5cee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:53:54.136471 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 764/1520] 10.222.0.52 () {58 vars in 1058 bytes} [Sun Sep 25 00:53:54 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:53:54.145086 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ff4575-fbcd-41ef-a159-6823a9f2e266 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:53:54.159472 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ff4575-fbcd-41ef-a159-6823a9f2e266 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:53:54.164421 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a663bb-c224-4078-a436-2b460ffd4958 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '11a80b55dcd444df901c8bf80f2a2c26', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:53:54.165027 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a663bb-c224-4078-a436-2b460ffd4958 service placement] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:53:54.165333 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a663bb-c224-4078-a436-2b460ffd4958 service placement] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:53:54.165601 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a663bb-c224-4078-a436-2b460ffd4958 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:53:54.175052 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7a663bb-c224-4078-a436-2b460ffd4958 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:53:54.176648 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7a663bb-c224-4078-a436-2b460ffd4958 service placement] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:53:54.188318 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 757/1521] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:53:54 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:53:56.952819 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed20583-63f5-46a6-b217-a58ed23eaeab None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:53:56.953464 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed20583-63f5-46a6-b217-a58ed23eaeab None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:53:56.953464 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed20583-63f5-46a6-b217-a58ed23eaeab None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:53:56.954634 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 765/1522] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 25 00:53:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:53:56.961199 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b149afb-a51d-49bc-ab1a-d4ec214afc81 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:53:56.961701 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b149afb-a51d-49bc-ab1a-d4ec214afc81 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:53:56.961812 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b149afb-a51d-49bc-ab1a-d4ec214afc81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:53:56.981480 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-3b149afb-a51d-49bc-ab1a-d4ec214afc81 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:53:56.983581 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-3b149afb-a51d-49bc-ab1a-d4ec214afc81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:53:56.991891 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 758/1523] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 25 00:53:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:53:57.154910 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed20583-63f5-46a6-b217-a58ed23eaeab None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:53:57.169017 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed20583-63f5-46a6-b217-a58ed23eaeab None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:53:57.175324 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd3542dd-2782-48fe-97b7-2ef416955ffd service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f142988a4dd14a5f96eacea45c046f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:53:57.175913 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3542dd-2782-48fe-97b7-2ef416955ffd service nova] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:53:57.176180 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3542dd-2782-48fe-97b7-2ef416955ffd service nova] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:53:57.176442 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3542dd-2782-48fe-97b7-2ef416955ffd service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:53:57.185520 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd3542dd-2782-48fe-97b7-2ef416955ffd service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:53:57.187094 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd3542dd-2782-48fe-97b7-2ef416955ffd service nova] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:53:57.196611 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 766/1524] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:53:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:53:58.281328 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c4e764-1c62-4ae1-9b78-4433a5e05170 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:53:58.281868 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c4e764-1c62-4ae1-9b78-4433a5e05170 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:53:58.281868 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c4e764-1c62-4ae1-9b78-4433a5e05170 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:53:58.283130 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 759/1525] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 25 00:53:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:53:58.289659 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe61b4c-8d45-47d6-997b-8d0e8f4c7fec None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:53:58.290135 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe61b4c-8d45-47d6-997b-8d0e8f4c7fec None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:53:58.290241 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe61b4c-8d45-47d6-997b-8d0e8f4c7fec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:53:58.309278 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-ffe61b4c-8d45-47d6-997b-8d0e8f4c7fec None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:53:58.311296 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-ffe61b4c-8d45-47d6-997b-8d0e8f4c7fec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:53:58.319044 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 767/1526] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 25 00:53:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 00:53:58.471963 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c4e764-1c62-4ae1-9b78-4433a5e05170 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:53:58.487326 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c4e764-1c62-4ae1-9b78-4433a5e05170 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:53:58.492909 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf9995db-37fe-4ed9-bbb0-ed54ec1d6831 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f142988a4dd14a5f96eacea45c046f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:53:58.493527 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf9995db-37fe-4ed9-bbb0-ed54ec1d6831 service nova] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:53:58.493802 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf9995db-37fe-4ed9-bbb0-ed54ec1d6831 service nova] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:53:58.494068 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf9995db-37fe-4ed9-bbb0-ed54ec1d6831 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:53:58.502955 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf9995db-37fe-4ed9-bbb0-ed54ec1d6831 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:53:58.504519 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf9995db-37fe-4ed9-bbb0-ed54ec1d6831 service nova] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:53:58.514733 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 760/1527] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:53:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:54:02.471219 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76e6529-1035-4854-8f38-aec1369a2364 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:54:02.471743 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76e6529-1035-4854-8f38-aec1369a2364 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:54:02.471743 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76e6529-1035-4854-8f38-aec1369a2364 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:54:02.473012 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 768/1528] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 25 00:54:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:54:02.478352 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020ce346-8af3-4aca-89de-d4da50761b29 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:54:02.478789 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020ce346-8af3-4aca-89de-d4da50761b29 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:54:02.478894 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020ce346-8af3-4aca-89de-d4da50761b29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:54:02.500351 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-020ce346-8af3-4aca-89de-d4da50761b29 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:54:02.501566 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-020ce346-8af3-4aca-89de-d4da50761b29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:54:02.509445 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 761/1529] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 25 00:54:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:54:02.805781 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e76e6529-1035-4854-8f38-aec1369a2364 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:54:02.807479 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-e76e6529-1035-4854-8f38-aec1369a2364 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:54:02.842013 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:54:02.891243 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e76e6529-1035-4854-8f38-aec1369a2364 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:54:02.896474 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363b6f82-786b-4de9-a2d7-ec202250548f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:54:02.897083 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363b6f82-786b-4de9-a2d7-ec202250548f service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:54:02.897378 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363b6f82-786b-4de9-a2d7-ec202250548f service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:54:02.897645 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363b6f82-786b-4de9-a2d7-ec202250548f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:54:02.906999 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-363b6f82-786b-4de9-a2d7-ec202250548f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:54:02.908412 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-363b6f82-786b-4de9-a2d7-ec202250548f service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:54:02.917790 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 769/1530] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:54:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:54:05.105639 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b73eebc-e0d9-420c-8b68-33c1961f8feb None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:54:05.106205 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b73eebc-e0d9-420c-8b68-33c1961f8feb None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:54:05.106205 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b73eebc-e0d9-420c-8b68-33c1961f8feb None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:54:05.107627 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 762/1531] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 25 00:54:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:54:05.113799 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5f934f-7842-46ec-aa8e-1bc5ee00324b None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:54:05.114302 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5f934f-7842-46ec-aa8e-1bc5ee00324b None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:54:05.114421 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5f934f-7842-46ec-aa8e-1bc5ee00324b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:54:05.134332 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-3f5f934f-7842-46ec-aa8e-1bc5ee00324b None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:54:05.135541 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-3f5f934f-7842-46ec-aa8e-1bc5ee00324b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:54:05.143763 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 770/1532] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 25 00:54:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:54:05.433909 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b73eebc-e0d9-420c-8b68-33c1961f8feb None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:54:05.444567 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b73eebc-e0d9-420c-8b68-33c1961f8feb None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:54:05.448042 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb8bf333-2093-49ac-9674-528ea22add05 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:54:05.448629 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb8bf333-2093-49ac-9674-528ea22add05 service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:54:05.448898 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb8bf333-2093-49ac-9674-528ea22add05 service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:54:05.449161 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb8bf333-2093-49ac-9674-528ea22add05 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:54:05.459028 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb8bf333-2093-49ac-9674-528ea22add05 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:54:05.460430 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb8bf333-2093-49ac-9674-528ea22add05 service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:54:05.471325 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 763/1533] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:54:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:54:09.246630 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-526a2b83-adb0-4f1e-b841-28af7610eac7 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:54:09.247861 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-526a2b83-adb0-4f1e-b841-28af7610eac7 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:54:09.247861 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-526a2b83-adb0-4f1e-b841-28af7610eac7 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:54:09.249414 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 771/1534] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 25 00:54:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:54:09.255083 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f7fc8b-e0de-4af3-8d0d-7a9896d969d3 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:54:09.255587 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f7fc8b-e0de-4af3-8d0d-7a9896d969d3 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:54:09.255587 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f7fc8b-e0de-4af3-8d0d-7a9896d969d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:54:09.276103 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-48f7fc8b-e0de-4af3-8d0d-7a9896d969d3 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:54:09.277319 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-48f7fc8b-e0de-4af3-8d0d-7a9896d969d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:54:09.285811 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 764/1535] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 25 00:54:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:54:09.593343 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-526a2b83-adb0-4f1e-b841-28af7610eac7 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:54:09.595047 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-526a2b83-adb0-4f1e-b841-28af7610eac7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:54:09.677962 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-526a2b83-adb0-4f1e-b841-28af7610eac7 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:54:09.681676 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e412eab-8dd2-45de-92d6-276c47fa3c4f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:54:09.682287 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e412eab-8dd2-45de-92d6-276c47fa3c4f service cinder] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:54:09.682570 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e412eab-8dd2-45de-92d6-276c47fa3c4f service cinder] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:54:09.682841 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e412eab-8dd2-45de-92d6-276c47fa3c4f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:54:09.693176 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e412eab-8dd2-45de-92d6-276c47fa3c4f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:54:09.694664 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e412eab-8dd2-45de-92d6-276c47fa3c4f service cinder] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:54:09.705569 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 772/1536] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:54:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:54:11.919571 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee5c25a-376c-4909-b89e-bccf27a6c3e0 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:54:11.920140 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee5c25a-376c-4909-b89e-bccf27a6c3e0 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:54:11.920140 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee5c25a-376c-4909-b89e-bccf27a6c3e0 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:54:11.921322 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 765/1537] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 25 00:54:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 00:54:11.926893 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2de5128-d720-4b74-a380-c5a6adfbb23d None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:54:11.927330 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2de5128-d720-4b74-a380-c5a6adfbb23d None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:54:11.927442 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2de5128-d720-4b74-a380-c5a6adfbb23d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:54:11.947835 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-c2de5128-d720-4b74-a380-c5a6adfbb23d None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:54:11.949022 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-c2de5128-d720-4b74-a380-c5a6adfbb23d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:54:11.958405 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 773/1538] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 25 00:54:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:54:12.275329 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee5c25a-376c-4909-b89e-bccf27a6c3e0 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:54:12.277001 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-cee5c25a-376c-4909-b89e-bccf27a6c3e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:54:12.290212 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee5c25a-376c-4909-b89e-bccf27a6c3e0 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:54:12.294262 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-608b6552-6188-4192-8620-98061d0faa8b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:54:12.295196 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608b6552-6188-4192-8620-98061d0faa8b service cinder] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:54:12.295475 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608b6552-6188-4192-8620-98061d0faa8b service cinder] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:54:12.295738 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608b6552-6188-4192-8620-98061d0faa8b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:54:12.305336 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-608b6552-6188-4192-8620-98061d0faa8b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:54:12.306805 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-608b6552-6188-4192-8620-98061d0faa8b service cinder] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:54:12.317176 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 766/1539] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:54:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:54:21.942969 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=63 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:54:22.313317 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=37 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:55:37.309308 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dedf5858-c06c-42ca-87d9-26cbcd6628b2 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:37.426735 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dedf5858-c06c-42ca-87d9-26cbcd6628b2 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:37.426735 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dedf5858-c06c-42ca-87d9-26cbcd6628b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:37.426735 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-dedf5858-c06c-42ca-87d9-26cbcd6628b2 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:55:37.426735 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-dedf5858-c06c-42ca-87d9-26cbcd6628b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:37.426735 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 774/1540] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:55:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:55:37.427202 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634c685d-e5ed-4e11-b3b1-f0d7ce12ff0d None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:37.427202 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634c685d-e5ed-4e11-b3b1-f0d7ce12ff0d None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:37.427202 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634c685d-e5ed-4e11-b3b1-f0d7ce12ff0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:37.427202 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-634c685d-e5ed-4e11-b3b1-f0d7ce12ff0d None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:55:37.427202 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-634c685d-e5ed-4e11-b3b1-f0d7ce12ff0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:37.427202 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 767/1541] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:55:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:55:37.428055 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dedf5858-c06c-42ca-87d9-26cbcd6628b2 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:37.428146 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788c8ec1-7bf9-434c-9eb2-0d02ee9b16da None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:37.428146 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788c8ec1-7bf9-434c-9eb2-0d02ee9b16da None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:37.428146 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788c8ec1-7bf9-434c-9eb2-0d02ee9b16da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:37.428411 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dedf5858-c06c-42ca-87d9-26cbcd6628b2 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:37.428411 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d335fb-8190-40f8-920e-2ed56683d457 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:37.428411 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d335fb-8190-40f8-920e-2ed56683d457 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:37.428411 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d335fb-8190-40f8-920e-2ed56683d457 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:37.428741 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d335fb-8190-40f8-920e-2ed56683d457 admin admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:37.428741 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8d335fb-8190-40f8-920e-2ed56683d457 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:55:37.428741 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8d335fb-8190-40f8-920e-2ed56683d457 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:37.428741 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8d335fb-8190-40f8-920e-2ed56683d457 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:37.439583 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 775/1542] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 00:55:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:55:37.448760 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d335fb-8190-40f8-920e-2ed56683d457 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:37.454349 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-788c8ec1-7bf9-434c-9eb2-0d02ee9b16da None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:55:37.455570 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-788c8ec1-7bf9-434c-9eb2-0d02ee9b16da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:37.464812 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d335fb-8190-40f8-920e-2ed56683d457 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:37.470080 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1ea06c-c190-4a34-8ec1-46af356754f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:37.470824 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1ea06c-c190-4a34-8ec1-46af356754f4 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:37.471110 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1ea06c-c190-4a34-8ec1-46af356754f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:37.471483 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1ea06c-c190-4a34-8ec1-46af356754f4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:37.472415 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e1ea06c-c190-4a34-8ec1-46af356754f4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:37.473597 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: WARNING py.warnings [None req-9e1ea06c-c190-4a34-8ec1-46af356754f4 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 25 00:55:37.473597 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: warnings.warn(msg) Sep 25 00:55:37.473597 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: Sep 25 00:55:37.474944 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e1ea06c-c190-4a34-8ec1-46af356754f4 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:37.490901 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 768/1543] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:55:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 00:55:37.492455 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0eaa7c-238e-41cb-88cd-473681c606a7 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:37.492947 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0eaa7c-238e-41cb-88cd-473681c606a7 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:37.493534 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0eaa7c-238e-41cb-88cd-473681c606a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:37.541711 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 776/1544] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:55:37 2022] POST /identity/v3/projects => generated 385 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:55:37.543394 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-6e0eaa7c-238e-41cb-88cd-473681c606a7 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:55:37.545564 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-6e0eaa7c-238e-41cb-88cd-473681c606a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:37.552618 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08646e7-e49b-403c-aa5f-329f58f36978 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:37.552618 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08646e7-e49b-403c-aa5f-329f58f36978 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:37.552618 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08646e7-e49b-403c-aa5f-329f58f36978 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:37.579796 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-a08646e7-e49b-403c-aa5f-329f58f36978 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:55:37.586190 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-a08646e7-e49b-403c-aa5f-329f58f36978 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:37.671090 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 777/1545] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:55:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:55:37.671950 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08646e7-e49b-403c-aa5f-329f58f36978 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:37.683822 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08646e7-e49b-403c-aa5f-329f58f36978 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:37.689016 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93330e5c-e306-45ec-a52a-9a07494521be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:37.689724 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93330e5c-e306-45ec-a52a-9a07494521be admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:37.690190 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93330e5c-e306-45ec-a52a-9a07494521be admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:37.690468 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93330e5c-e306-45ec-a52a-9a07494521be admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:37.691269 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93330e5c-e306-45ec-a52a-9a07494521be admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:37.693888 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93330e5c-e306-45ec-a52a-9a07494521be admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:37.695985 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 769/1546] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:55:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:55:37.697845 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4276b4a7-1cb1-40aa-9899-2ab1a3d4cb76 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:37.698371 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4276b4a7-1cb1-40aa-9899-2ab1a3d4cb76 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:37.698851 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4276b4a7-1cb1-40aa-9899-2ab1a3d4cb76 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:37.718708 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 778/1547] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:55:37 2022] POST /identity/v3/users => generated 293 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:55:37.721358 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57adddf-141e-475d-81ed-7a29b2b47fa1 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:37.721650 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57adddf-141e-475d-81ed-7a29b2b47fa1 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:37.722421 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57adddf-141e-475d-81ed-7a29b2b47fa1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:37.724570 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-4276b4a7-1cb1-40aa-9899-2ab1a3d4cb76 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:55:37.725867 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-4276b4a7-1cb1-40aa-9899-2ab1a3d4cb76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:37.736203 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 770/1548] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:55:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:55:37.737042 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4276b4a7-1cb1-40aa-9899-2ab1a3d4cb76 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:37.747142 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-e57adddf-141e-475d-81ed-7a29b2b47fa1 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:55:37.748377 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-e57adddf-141e-475d-81ed-7a29b2b47fa1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:37.749693 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4276b4a7-1cb1-40aa-9899-2ab1a3d4cb76 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:37.755471 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6397adce-183b-449d-92d4-6e30c2b30b63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:37.756129 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6397adce-183b-449d-92d4-6e30c2b30b63 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:37.756409 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6397adce-183b-449d-92d4-6e30c2b30b63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:37.756678 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6397adce-183b-449d-92d4-6e30c2b30b63 admin admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:37.757388 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6397adce-183b-449d-92d4-6e30c2b30b63 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:55:37.757695 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6397adce-183b-449d-92d4-6e30c2b30b63 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:37.757818 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 779/1549] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:55:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:55:37.758714 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: WARNING py.warnings [None req-6397adce-183b-449d-92d4-6e30c2b30b63 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 25 00:55:37.758714 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: warnings.warn(msg) Sep 25 00:55:37.758714 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: Sep 25 00:55:37.759172 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-016fc936-6377-4182-8e80-8132ee15f2c1 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:37.759466 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-016fc936-6377-4182-8e80-8132ee15f2c1 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:37.759721 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-016fc936-6377-4182-8e80-8132ee15f2c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:37.760258 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6397adce-183b-449d-92d4-6e30c2b30b63 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:37.776215 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 771/1550] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 00:55:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:55:37.777553 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70098023-6023-4d02-a266-eb5742129043 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:37.777838 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70098023-6023-4d02-a266-eb5742129043 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:37.778107 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70098023-6023-4d02-a266-eb5742129043 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:37.782387 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-016fc936-6377-4182-8e80-8132ee15f2c1 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:55:37.784104 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-016fc936-6377-4182-8e80-8132ee15f2c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:37.793969 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 780/1551] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:55:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:55:37.794821 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-016fc936-6377-4182-8e80-8132ee15f2c1 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:37.806242 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-016fc936-6377-4182-8e80-8132ee15f2c1 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:37.809639 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-70098023-6023-4d02-a266-eb5742129043 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:55:37.811466 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-70098023-6023-4d02-a266-eb5742129043 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:37.811812 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a742a936-df65-46f5-8bce-262eca9e3789 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:37.812745 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a742a936-df65-46f5-8bce-262eca9e3789 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:37.813027 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a742a936-df65-46f5-8bce-262eca9e3789 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:37.813326 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a742a936-df65-46f5-8bce-262eca9e3789 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:37.814287 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a742a936-df65-46f5-8bce-262eca9e3789 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:37.816220 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a742a936-df65-46f5-8bce-262eca9e3789 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:37.821932 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 772/1552] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:55:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:55:37.823655 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-877f3baf-d636-476b-a879-619dcf575703 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:37.824160 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-877f3baf-d636-476b-a879-619dcf575703 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:37.824628 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-877f3baf-d636-476b-a879-619dcf575703 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:37.828709 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 781/1553] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:55:37 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:55:37.830243 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73989bc-c381-4b13-b646-5681cc764b2d None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:37.830468 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73989bc-c381-4b13-b646-5681cc764b2d None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:37.830768 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73989bc-c381-4b13-b646-5681cc764b2d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:37.846222 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-877f3baf-d636-476b-a879-619dcf575703 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:55:37.847395 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-877f3baf-d636-476b-a879-619dcf575703 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:37.852006 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-c73989bc-c381-4b13-b646-5681cc764b2d None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:55:37.853808 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-c73989bc-c381-4b13-b646-5681cc764b2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:37.856308 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 773/1554] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:55:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:55:37.856888 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-877f3baf-d636-476b-a879-619dcf575703 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:37.862044 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 782/1555] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:55:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:55:37.863502 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1901fa-0f02-43d5-b249-a5216bf95d93 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:37.863782 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1901fa-0f02-43d5-b249-a5216bf95d93 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:37.864205 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1901fa-0f02-43d5-b249-a5216bf95d93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:37.866633 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-877f3baf-d636-476b-a879-619dcf575703 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:37.871534 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db1366d-e3dc-46b4-998d-682bc89637db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:37.872128 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db1366d-e3dc-46b4-998d-682bc89637db admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:37.872409 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db1366d-e3dc-46b4-998d-682bc89637db admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:37.872659 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db1366d-e3dc-46b4-998d-682bc89637db admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:37.873301 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8db1366d-e3dc-46b4-998d-682bc89637db admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:37.875172 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8db1366d-e3dc-46b4-998d-682bc89637db admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:37.883551 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-9c1901fa-0f02-43d5-b249-a5216bf95d93 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:55:37.884784 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-9c1901fa-0f02-43d5-b249-a5216bf95d93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:37.892783 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 783/1556] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:55:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:55:37.894032 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6b36a3-8f8d-47fd-9288-e56b56efcb51 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:37.894311 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6b36a3-8f8d-47fd-9288-e56b56efcb51 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:37.894581 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6b36a3-8f8d-47fd-9288-e56b56efcb51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:37.910895 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 774/1557] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:55:37 2022] POST /identity/v3/projects => generated 383 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:55:37.910895 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db1366d-e3dc-46b4-998d-682bc89637db admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:37.917967 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-1b6b36a3-8f8d-47fd-9288-e56b56efcb51 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:55:37.919169 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-1b6b36a3-8f8d-47fd-9288-e56b56efcb51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:37.969237 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:55:37.971812 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db1366d-e3dc-46b4-998d-682bc89637db admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:37.976655 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37e5546c-b89f-436a-901d-8dddbb0b4163 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:37.977269 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e5546c-b89f-436a-901d-8dddbb0b4163 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:37.977560 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e5546c-b89f-436a-901d-8dddbb0b4163 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:37.978474 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e5546c-b89f-436a-901d-8dddbb0b4163 admin admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:37.979066 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37e5546c-b89f-436a-901d-8dddbb0b4163 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:55:37.979405 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37e5546c-b89f-436a-901d-8dddbb0b4163 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:37.980997 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37e5546c-b89f-436a-901d-8dddbb0b4163 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:37.994764 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 784/1558] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:55:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:55:37.994764 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d24159-1b31-4f7d-a466-b4a54fc31ffd None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:37.994764 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d24159-1b31-4f7d-a466-b4a54fc31ffd None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:37.994764 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d24159-1b31-4f7d-a466-b4a54fc31ffd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:37.995598 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 775/1559] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 00:55:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:55:37.995598 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37e5546c-b89f-436a-901d-8dddbb0b4163 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:38.005113 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37e5546c-b89f-436a-901d-8dddbb0b4163 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:38.010892 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7900b05f-7969-479e-b5bc-c58a2af6b6cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:38.011487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7900b05f-7969-479e-b5bc-c58a2af6b6cb admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:38.011760 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7900b05f-7969-479e-b5bc-c58a2af6b6cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:38.012027 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7900b05f-7969-479e-b5bc-c58a2af6b6cb admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:38.012684 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7900b05f-7969-479e-b5bc-c58a2af6b6cb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:38.014293 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7900b05f-7969-479e-b5bc-c58a2af6b6cb admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:38.015706 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-23d24159-1b31-4f7d-a466-b4a54fc31ffd None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:55:38.016910 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-23d24159-1b31-4f7d-a466-b4a54fc31ffd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:38.024765 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 785/1560] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:55:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:55:38.029433 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 776/1561] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:55:37 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:55:38.029433 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7900b05f-7969-479e-b5bc-c58a2af6b6cb admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:38.029671 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23d24159-1b31-4f7d-a466-b4a54fc31ffd None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:38.038474 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23d24159-1b31-4f7d-a466-b4a54fc31ffd None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:38.039289 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7900b05f-7969-479e-b5bc-c58a2af6b6cb admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:38.043055 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35468697-fe8c-4cd7-a5c6-4d621b69bfc9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:38.043651 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35468697-fe8c-4cd7-a5c6-4d621b69bfc9 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:38.043926 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35468697-fe8c-4cd7-a5c6-4d621b69bfc9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:38.044024 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143fe6c3-fa68-4340-98e3-20c5b4cd8c00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:38.044196 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35468697-fe8c-4cd7-a5c6-4d621b69bfc9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:38.044584 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143fe6c3-fa68-4340-98e3-20c5b4cd8c00 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:38.044839 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35468697-fe8c-4cd7-a5c6-4d621b69bfc9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:55:38.044960 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143fe6c3-fa68-4340-98e3-20c5b4cd8c00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:38.045104 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143fe6c3-fa68-4340-98e3-20c5b4cd8c00 admin admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:38.045199 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35468697-fe8c-4cd7-a5c6-4d621b69bfc9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:38.045758 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-143fe6c3-fa68-4340-98e3-20c5b4cd8c00 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:55:38.046051 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-143fe6c3-fa68-4340-98e3-20c5b4cd8c00 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:38.046745 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35468697-fe8c-4cd7-a5c6-4d621b69bfc9 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:38.047634 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-143fe6c3-fa68-4340-98e3-20c5b4cd8c00 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:38.059271 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 786/1562] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 00:55:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:55:38.059928 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35468697-fe8c-4cd7-a5c6-4d621b69bfc9 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:38.060047 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 777/1563] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 00:55:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:55:38.060754 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143fe6c3-fa68-4340-98e3-20c5b4cd8c00 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:38.065670 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:55:38.076103 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143fe6c3-fa68-4340-98e3-20c5b4cd8c00 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:38.079488 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35468697-fe8c-4cd7-a5c6-4d621b69bfc9 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:38.083764 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f0af0b0-3f2b-4cde-8ffa-902a5dbcba9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:38.084070 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f83ca387-163d-4c42-b06d-6a1b00c0c9eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:38.084333 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0af0b0-3f2b-4cde-8ffa-902a5dbcba9b admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:38.084626 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0af0b0-3f2b-4cde-8ffa-902a5dbcba9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:38.084755 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83ca387-163d-4c42-b06d-6a1b00c0c9eb admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:38.084855 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0af0b0-3f2b-4cde-8ffa-902a5dbcba9b admin admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:38.084951 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83ca387-163d-4c42-b06d-6a1b00c0c9eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:38.085216 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83ca387-163d-4c42-b06d-6a1b00c0c9eb admin admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:38.085525 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f0af0b0-3f2b-4cde-8ffa-902a5dbcba9b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:38.085883 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f83ca387-163d-4c42-b06d-6a1b00c0c9eb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:55:38.086181 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f83ca387-163d-4c42-b06d-6a1b00c0c9eb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:38.086522 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: WARNING py.warnings [None req-4f0af0b0-3f2b-4cde-8ffa-902a5dbcba9b admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 25 00:55:38.086522 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: warnings.warn(msg) Sep 25 00:55:38.086522 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: Sep 25 00:55:38.088001 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f83ca387-163d-4c42-b06d-6a1b00c0c9eb admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:38.088274 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f0af0b0-3f2b-4cde-8ffa-902a5dbcba9b admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:38.101680 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 787/1564] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 00:55:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:55:38.103220 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc5c713-a8f6-42c8-b974-67b0d3c818b6 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:38.103712 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc5c713-a8f6-42c8-b974-67b0d3c818b6 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:38.104155 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc5c713-a8f6-42c8-b974-67b0d3c818b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:38.116543 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 778/1565] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:55:38 2022] POST /identity/v3/users => generated 292 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:55:38.117205 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f0af0b0-3f2b-4cde-8ffa-902a5dbcba9b admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:38.127176 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-fdc5c713-a8f6-42c8-b974-67b0d3c818b6 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:55:38.128395 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-fdc5c713-a8f6-42c8-b974-67b0d3c818b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:38.129866 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f0af0b0-3f2b-4cde-8ffa-902a5dbcba9b admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:38.135193 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f300f4e-2cbf-47f9-9aa5-f60f2d291724 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:38.135816 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f300f4e-2cbf-47f9-9aa5-f60f2d291724 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:38.136088 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f300f4e-2cbf-47f9-9aa5-f60f2d291724 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:38.136352 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f300f4e-2cbf-47f9-9aa5-f60f2d291724 admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:38.136942 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 788/1566] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:55:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:55:38.137057 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f300f4e-2cbf-47f9-9aa5-f60f2d291724 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:38.137727 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc5c713-a8f6-42c8-b974-67b0d3c818b6 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:38.139111 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f300f4e-2cbf-47f9-9aa5-f60f2d291724 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:38.147035 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc5c713-a8f6-42c8-b974-67b0d3c818b6 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:38.151745 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-716c7b6c-7bff-41f6-a901-5bc55fcfb5fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:38.152431 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-716c7b6c-7bff-41f6-a901-5bc55fcfb5fc admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:38.152707 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-716c7b6c-7bff-41f6-a901-5bc55fcfb5fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:38.152975 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-716c7b6c-7bff-41f6-a901-5bc55fcfb5fc admin admin] PATH_INFO: `/v3/projects/1bc1cab13abc43779d57421107671bdf/users/782f421af38b4c82a38ffc6ec1758b67/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:38.168422 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 779/1567] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:55:38 2022] POST /identity/v3/projects => generated 385 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:55:38.169858 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-716c7b6c-7bff-41f6-a901-5bc55fcfb5fc admin admin] RBAC: Authorizing `identity:create_grant(project_id=1bc1cab13abc43779d57421107671bdf, user_id=782f421af38b4c82a38ffc6ec1758b67, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:38.170022 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0373a0-f463-4cf7-82a9-09083d4bf715 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:38.170264 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0373a0-f463-4cf7-82a9-09083d4bf715 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:38.170587 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0373a0-f463-4cf7-82a9-09083d4bf715 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:38.170685 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: WARNING py.warnings [None req-716c7b6c-7bff-41f6-a901-5bc55fcfb5fc admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 25 00:55:38.170685 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: warnings.warn(msg) Sep 25 00:55:38.170685 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: Sep 25 00:55:38.172526 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-716c7b6c-7bff-41f6-a901-5bc55fcfb5fc admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:38.187140 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 789/1568] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:55:38 2022] PUT /identity/v3/projects/1bc1cab13abc43779d57421107671bdf/users/782f421af38b4c82a38ffc6ec1758b67/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:55:38.187931 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-716c7b6c-7bff-41f6-a901-5bc55fcfb5fc admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:38.192443 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-2f0373a0-f463-4cf7-82a9-09083d4bf715 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:55:38.193711 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-2f0373a0-f463-4cf7-82a9-09083d4bf715 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:38.254561 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-716c7b6c-7bff-41f6-a901-5bc55fcfb5fc admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:38.257863 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 780/1569] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:55:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:55:38.258673 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0373a0-f463-4cf7-82a9-09083d4bf715 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:38.263022 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a8ec33-9a03-4eb5-abf4-f30224acd582 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:38.263636 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a8ec33-9a03-4eb5-abf4-f30224acd582 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:38.263957 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a8ec33-9a03-4eb5-abf4-f30224acd582 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:38.264226 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a8ec33-9a03-4eb5-abf4-f30224acd582 admin admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:38.264859 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5a8ec33-9a03-4eb5-abf4-f30224acd582 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:38.266812 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5a8ec33-9a03-4eb5-abf4-f30224acd582 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:38.268914 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0373a0-f463-4cf7-82a9-09083d4bf715 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:38.273798 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdcb4f75-6a80-41f5-bfce-ae55f1ffa82e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:38.274384 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcb4f75-6a80-41f5-bfce-ae55f1ffa82e admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:38.274736 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcb4f75-6a80-41f5-bfce-ae55f1ffa82e admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:38.275212 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcb4f75-6a80-41f5-bfce-ae55f1ffa82e admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:38.275967 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdcb4f75-6a80-41f5-bfce-ae55f1ffa82e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:38.277882 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdcb4f75-6a80-41f5-bfce-ae55f1ffa82e admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:38.298030 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 790/1570] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:55:38 2022] POST /identity/v3/projects => generated 365 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:55:38.298030 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a8ec33-9a03-4eb5-abf4-f30224acd582 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:38.313442 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 781/1571] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:55:38 2022] POST /identity/v3/projects => generated 393 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:55:38.313442 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdcb4f75-6a80-41f5-bfce-ae55f1ffa82e admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:38.371048 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a8ec33-9a03-4eb5-abf4-f30224acd582 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:38.375651 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7299204c-617b-4473-8cb9-3168451e800e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:38.376254 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7299204c-617b-4473-8cb9-3168451e800e admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:38.376538 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7299204c-617b-4473-8cb9-3168451e800e admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:38.376796 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7299204c-617b-4473-8cb9-3168451e800e admin admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:38.377450 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7299204c-617b-4473-8cb9-3168451e800e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:38.378974 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdcb4f75-6a80-41f5-bfce-ae55f1ffa82e admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:38.379351 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7299204c-617b-4473-8cb9-3168451e800e admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:38.383632 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a5e1ea9-4e3f-4cb4-8851-98f4bbd46046 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:38.384881 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5e1ea9-4e3f-4cb4-8851-98f4bbd46046 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:38.385407 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5e1ea9-4e3f-4cb4-8851-98f4bbd46046 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:38.385684 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5e1ea9-4e3f-4cb4-8851-98f4bbd46046 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:38.386326 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a5e1ea9-4e3f-4cb4-8851-98f4bbd46046 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:38.387901 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a5e1ea9-4e3f-4cb4-8851-98f4bbd46046 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:38.398292 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 782/1572] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:55:38 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:55:38.399309 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a5e1ea9-4e3f-4cb4-8851-98f4bbd46046 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:38.410158 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a5e1ea9-4e3f-4cb4-8851-98f4bbd46046 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:38.413808 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 791/1573] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:55:38 2022] POST /identity/v3/projects => generated 379 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:55:38.414705 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7299204c-617b-4473-8cb9-3168451e800e admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:38.418965 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:55:38.444527 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:55:38.470415 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a69f06-0f5d-4632-8e78-9f3d6574b1d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:38.471075 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a69f06-0f5d-4632-8e78-9f3d6574b1d4 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:38.471354 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a69f06-0f5d-4632-8e78-9f3d6574b1d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:38.471619 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a69f06-0f5d-4632-8e78-9f3d6574b1d4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:38.472265 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24a69f06-0f5d-4632-8e78-9f3d6574b1d4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:55:38.472560 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24a69f06-0f5d-4632-8e78-9f3d6574b1d4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:38.474176 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24a69f06-0f5d-4632-8e78-9f3d6574b1d4 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:38.479378 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7299204c-617b-4473-8cb9-3168451e800e admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:38.484375 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e7770c-fed9-4298-9d1a-2cbc4d4896d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:38.485103 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e7770c-fed9-4298-9d1a-2cbc4d4896d9 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:38.485428 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e7770c-fed9-4298-9d1a-2cbc4d4896d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:38.485723 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e7770c-fed9-4298-9d1a-2cbc4d4896d9 admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:38.486355 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63e7770c-fed9-4298-9d1a-2cbc4d4896d9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:38.487371 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 783/1574] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 00:55:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:55:38.488695 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63e7770c-fed9-4298-9d1a-2cbc4d4896d9 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:38.488979 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18fe37c-0490-4a83-81bf-7421c1b26470 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:38.489279 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18fe37c-0490-4a83-81bf-7421c1b26470 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:38.489546 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18fe37c-0490-4a83-81bf-7421c1b26470 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:38.512699 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 792/1575] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:55:38 2022] POST /identity/v3/users => generated 293 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:55:38.514327 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e7770c-fed9-4298-9d1a-2cbc4d4896d9 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:38.524159 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-a18fe37c-0490-4a83-81bf-7421c1b26470 None None] MFA Rules not processed for user `782f421af38b4c82a38ffc6ec1758b67`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:55:38.524367 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e7770c-fed9-4298-9d1a-2cbc4d4896d9 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:38.525886 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-a18fe37c-0490-4a83-81bf-7421c1b26470 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:38.529502 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3899b65d-d601-4e69-925d-fa3853c61c70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:38.530105 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3899b65d-d601-4e69-925d-fa3853c61c70 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:38.530384 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3899b65d-d601-4e69-925d-fa3853c61c70 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:38.530656 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3899b65d-d601-4e69-925d-fa3853c61c70 admin admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:38.531331 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3899b65d-d601-4e69-925d-fa3853c61c70 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:55:38.531630 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3899b65d-d601-4e69-925d-fa3853c61c70 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:38.533219 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3899b65d-d601-4e69-925d-fa3853c61c70 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:38.546742 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 793/1576] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 00:55:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:55:38.547551 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3899b65d-d601-4e69-925d-fa3853c61c70 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:38.557843 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3899b65d-d601-4e69-925d-fa3853c61c70 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:38.562434 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841151a7-f507-481b-8574-cc932bf5b38f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:38.563019 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841151a7-f507-481b-8574-cc932bf5b38f admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:38.563291 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841151a7-f507-481b-8574-cc932bf5b38f admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:38.563555 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841151a7-f507-481b-8574-cc932bf5b38f admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:38.564170 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-841151a7-f507-481b-8574-cc932bf5b38f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:38.566093 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-841151a7-f507-481b-8574-cc932bf5b38f admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:38.589116 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 794/1577] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:55:38 2022] POST /identity/v3/users => generated 283 bytes in 42 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:55:38.589958 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841151a7-f507-481b-8574-cc932bf5b38f admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:38.596539 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 784/1578] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:55:38 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:55:38.600874 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a18fe37c-0490-4a83-81bf-7421c1b26470 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:38.604820 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841151a7-f507-481b-8574-cc932bf5b38f admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:38.610140 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9205851e-a730-4fa6-a629-dfa2d15f9d21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:38.610373 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a18fe37c-0490-4a83-81bf-7421c1b26470 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:38.610747 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9205851e-a730-4fa6-a629-dfa2d15f9d21 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:38.611019 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9205851e-a730-4fa6-a629-dfa2d15f9d21 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:38.611338 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9205851e-a730-4fa6-a629-dfa2d15f9d21 admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:38.611908 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9205851e-a730-4fa6-a629-dfa2d15f9d21 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:38.614152 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9205851e-a730-4fa6-a629-dfa2d15f9d21 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:38.615510 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6805c3f-89d1-4f2b-af87-d80229d18bec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:38.616246 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6805c3f-89d1-4f2b-af87-d80229d18bec admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:38.616705 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6805c3f-89d1-4f2b-af87-d80229d18bec admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:38.617006 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6805c3f-89d1-4f2b-af87-d80229d18bec admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:38.617614 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6805c3f-89d1-4f2b-af87-d80229d18bec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:38.619340 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6805c3f-89d1-4f2b-af87-d80229d18bec admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:38.631368 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 785/1579] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:55:38 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:55:38.632180 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6805c3f-89d1-4f2b-af87-d80229d18bec admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:38.641668 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 795/1580] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:55:38 2022] POST /identity/v3/users => generated 297 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:55:38.643084 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9205851e-a730-4fa6-a629-dfa2d15f9d21 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:38.644726 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6805c3f-89d1-4f2b-af87-d80229d18bec admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:38.650427 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d041b8a-024a-4fba-ada0-7c4e6f195156 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:38.651296 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d041b8a-024a-4fba-ada0-7c4e6f195156 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:38.651577 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d041b8a-024a-4fba-ada0-7c4e6f195156 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:38.651844 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d041b8a-024a-4fba-ada0-7c4e6f195156 admin admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:38.652498 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d041b8a-024a-4fba-ada0-7c4e6f195156 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:38.654324 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9205851e-a730-4fa6-a629-dfa2d15f9d21 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:38.654447 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d041b8a-024a-4fba-ada0-7c4e6f195156 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:38.659244 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af57974-d655-4bb2-844e-f2c521f67ccd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:38.659829 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af57974-d655-4bb2-844e-f2c521f67ccd admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:38.660097 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af57974-d655-4bb2-844e-f2c521f67ccd admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:38.660360 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af57974-d655-4bb2-844e-f2c521f67ccd admin admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:38.660978 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af57974-d655-4bb2-844e-f2c521f67ccd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:38.662923 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af57974-d655-4bb2-844e-f2c521f67ccd admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:38.678999 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 786/1581] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:55:38 2022] POST /identity/v3/users => generated 290 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:55:38.679959 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d041b8a-024a-4fba-ada0-7c4e6f195156 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:38.690595 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d041b8a-024a-4fba-ada0-7c4e6f195156 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:38.696947 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 796/1582] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:55:38 2022] POST /identity/v3/projects => generated 405 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:55:38.696947 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af57974-d655-4bb2-844e-f2c521f67ccd admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:38.698715 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed2040b-f6fa-4d71-b7cc-54f601b018d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:38.699355 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed2040b-f6fa-4d71-b7cc-54f601b018d5 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:38.699635 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed2040b-f6fa-4d71-b7cc-54f601b018d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:38.699913 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed2040b-f6fa-4d71-b7cc-54f601b018d5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:38.700556 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ed2040b-f6fa-4d71-b7cc-54f601b018d5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:38.702194 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ed2040b-f6fa-4d71-b7cc-54f601b018d5 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:38.713644 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 787/1583] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:55:38 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:55:38.714452 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed2040b-f6fa-4d71-b7cc-54f601b018d5 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:38.723946 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed2040b-f6fa-4d71-b7cc-54f601b018d5 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:38.762376 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af57974-d655-4bb2-844e-f2c521f67ccd admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:38.768314 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-068caaf0-1215-40bd-a805-98dddc16e495 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:38.769171 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068caaf0-1215-40bd-a805-98dddc16e495 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:38.769595 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068caaf0-1215-40bd-a805-98dddc16e495 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:38.769870 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068caaf0-1215-40bd-a805-98dddc16e495 admin admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:38.770728 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-068caaf0-1215-40bd-a805-98dddc16e495 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:38.772971 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-068caaf0-1215-40bd-a805-98dddc16e495 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:38.782491 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b391d39-3e04-4e06-91bc-afdd2637ad75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:38.783090 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b391d39-3e04-4e06-91bc-afdd2637ad75 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:38.783380 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b391d39-3e04-4e06-91bc-afdd2637ad75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:38.783634 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b391d39-3e04-4e06-91bc-afdd2637ad75 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:38.784264 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b391d39-3e04-4e06-91bc-afdd2637ad75 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:38.785888 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b391d39-3e04-4e06-91bc-afdd2637ad75 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:38.797127 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 788/1584] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:55:38 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:55:38.798408 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a1c5a6d-0ede-4d2c-8967-d6f09004bfd4 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:38.798690 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a1c5a6d-0ede-4d2c-8967-d6f09004bfd4 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:38.798959 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a1c5a6d-0ede-4d2c-8967-d6f09004bfd4 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:38.800392 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 789/1585] 10.222.0.52 () {54 vars in 961 bytes} [Sun Sep 25 00:55:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:55:38.801079 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a1c5a6d-0ede-4d2c-8967-d6f09004bfd4 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:38.809317 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 797/1586] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:55:38 2022] POST /identity/v3/projects => generated 373 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:55:38.809317 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-068caaf0-1215-40bd-a805-98dddc16e495 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:38.818001 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=89,keystone:INSERT=11 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:55:38.824730 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:55:38.871340 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a1c5a6d-0ede-4d2c-8967-d6f09004bfd4 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:38.875455 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-068caaf0-1215-40bd-a805-98dddc16e495 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:38.876194 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7e3af7-fc56-4b18-b718-5819afc6d4d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:38.876855 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7e3af7-fc56-4b18-b718-5819afc6d4d8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:38.877225 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7e3af7-fc56-4b18-b718-5819afc6d4d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:38.877488 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7e3af7-fc56-4b18-b718-5819afc6d4d8 admin admin] PATH_INFO: `/v3/projects/f08425c00f0c49ddbe42b21ddbc7ff3f/users/96386a16f0b54b7c94d32bd948caf118/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:38.880693 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21554b94-7a20-4612-bad9-4421b7002e01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:38.881319 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21554b94-7a20-4612-bad9-4421b7002e01 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:38.881661 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21554b94-7a20-4612-bad9-4421b7002e01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:38.881879 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21554b94-7a20-4612-bad9-4421b7002e01 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:38.882645 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21554b94-7a20-4612-bad9-4421b7002e01 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:38.884787 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21554b94-7a20-4612-bad9-4421b7002e01 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:38.892923 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb7e3af7-fc56-4b18-b718-5819afc6d4d8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f08425c00f0c49ddbe42b21ddbc7ff3f, user_id=96386a16f0b54b7c94d32bd948caf118, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:38.895207 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 798/1587] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:55:38 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:55:38.895983 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21554b94-7a20-4612-bad9-4421b7002e01 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:38.896126 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb7e3af7-fc56-4b18-b718-5819afc6d4d8 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:38.907417 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21554b94-7a20-4612-bad9-4421b7002e01 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:38.910539 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 790/1588] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:55:38 2022] PUT /identity/v3/projects/f08425c00f0c49ddbe42b21ddbc7ff3f/users/96386a16f0b54b7c94d32bd948caf118/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:55:38.911525 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7e3af7-fc56-4b18-b718-5819afc6d4d8 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:38.966700 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51af0aa1-e9bf-47eb-a5b4-4fd541d32929 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:38.967307 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51af0aa1-e9bf-47eb-a5b4-4fd541d32929 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:38.967584 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51af0aa1-e9bf-47eb-a5b4-4fd541d32929 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:38.967851 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51af0aa1-e9bf-47eb-a5b4-4fd541d32929 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:38.968490 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51af0aa1-e9bf-47eb-a5b4-4fd541d32929 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:38.970124 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51af0aa1-e9bf-47eb-a5b4-4fd541d32929 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:38.973659 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7e3af7-fc56-4b18-b718-5819afc6d4d8 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:38.978476 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a14afc-e952-4a7e-89a7-79481df5660d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:38.979071 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a14afc-e952-4a7e-89a7-79481df5660d admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:38.979346 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a14afc-e952-4a7e-89a7-79481df5660d admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:38.979612 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a14afc-e952-4a7e-89a7-79481df5660d admin admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:38.980238 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a14afc-e952-4a7e-89a7-79481df5660d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:38.983329 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 799/1589] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:55:38 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:55:38.983329 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51af0aa1-e9bf-47eb-a5b4-4fd541d32929 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:38.983543 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a14afc-e952-4a7e-89a7-79481df5660d admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:38.991466 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51af0aa1-e9bf-47eb-a5b4-4fd541d32929 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:38.996480 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb2d081d-bbaa-4d30-ad2a-0cf6f2e99ab5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:38.997392 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2d081d-bbaa-4d30-ad2a-0cf6f2e99ab5 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:38.997684 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2d081d-bbaa-4d30-ad2a-0cf6f2e99ab5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:38.997957 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2d081d-bbaa-4d30-ad2a-0cf6f2e99ab5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:38.998709 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb2d081d-bbaa-4d30-ad2a-0cf6f2e99ab5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:39.000411 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb2d081d-bbaa-4d30-ad2a-0cf6f2e99ab5 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:39.005943 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 791/1590] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:55:38 2022] POST /identity/v3/users => generated 303 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:55:39.006764 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a14afc-e952-4a7e-89a7-79481df5660d admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:39.012436 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 800/1591] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:55:38 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:55:39.012436 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f5cebe-3712-408f-bfb4-fbf6e12bc5ee None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:39.012436 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f5cebe-3712-408f-bfb4-fbf6e12bc5ee None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:39.012436 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f5cebe-3712-408f-bfb4-fbf6e12bc5ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:39.020029 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a14afc-e952-4a7e-89a7-79481df5660d admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:39.024978 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8c5ad6-84b0-4189-9d07-87def51812ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:39.025707 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8c5ad6-84b0-4189-9d07-87def51812ef admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:39.026035 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8c5ad6-84b0-4189-9d07-87def51812ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:39.026309 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8c5ad6-84b0-4189-9d07-87def51812ef admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:39.026975 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8c5ad6-84b0-4189-9d07-87def51812ef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:39.028737 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8c5ad6-84b0-4189-9d07-87def51812ef admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:39.040687 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 792/1592] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:55:39 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:55:39.041731 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8c5ad6-84b0-4189-9d07-87def51812ef admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:39.051913 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8c5ad6-84b0-4189-9d07-87def51812ef admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:39.053217 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-23f5cebe-3712-408f-bfb4-fbf6e12bc5ee None None] MFA Rules not processed for user `f142988a4dd14a5f96eacea45c046f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:55:39.054483 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-23f5cebe-3712-408f-bfb4-fbf6e12bc5ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:39.057287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4bbcf87-2c58-4141-80dd-e478a41cc3b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:39.057893 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4bbcf87-2c58-4141-80dd-e478a41cc3b5 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:39.058164 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4bbcf87-2c58-4141-80dd-e478a41cc3b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:39.058433 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4bbcf87-2c58-4141-80dd-e478a41cc3b5 admin admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:39.059054 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4bbcf87-2c58-4141-80dd-e478a41cc3b5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:39.060951 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4bbcf87-2c58-4141-80dd-e478a41cc3b5 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:39.082984 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 793/1593] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:55:39 2022] POST /identity/v3/users => generated 287 bytes in 42 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:55:39.084014 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4bbcf87-2c58-4141-80dd-e478a41cc3b5 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:39.093794 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4bbcf87-2c58-4141-80dd-e478a41cc3b5 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:39.098768 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df43bad1-eba3-4631-9c2f-053054dde96d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:39.099367 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df43bad1-eba3-4631-9c2f-053054dde96d admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:39.099638 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df43bad1-eba3-4631-9c2f-053054dde96d admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:39.099897 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df43bad1-eba3-4631-9c2f-053054dde96d admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:39.100565 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df43bad1-eba3-4631-9c2f-053054dde96d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:39.102146 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df43bad1-eba3-4631-9c2f-053054dde96d admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:39.122431 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 794/1594] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:55:39 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:55:39.122431 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a451153-8150-4e9a-8d01-7efb674857d8 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:39.122431 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a451153-8150-4e9a-8d01-7efb674857d8 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:39.122431 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a451153-8150-4e9a-8d01-7efb674857d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:39.134361 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-5a451153-8150-4e9a-8d01-7efb674857d8 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:55:39.135468 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-5a451153-8150-4e9a-8d01-7efb674857d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:39.135994 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 801/1595] 10.222.0.52 () {58 vars in 1058 bytes} [Sun Sep 25 00:55:39 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:55:39.136834 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f5cebe-3712-408f-bfb4-fbf6e12bc5ee None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:39.143338 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 795/1596] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:55:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:55:39.144220 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a451153-8150-4e9a-8d01-7efb674857d8 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:39.147581 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f5cebe-3712-408f-bfb4-fbf6e12bc5ee None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:39.153131 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d05ddf3-71a5-4516-b208-9e72a71efae4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:39.153877 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d05ddf3-71a5-4516-b208-9e72a71efae4 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:39.154207 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d05ddf3-71a5-4516-b208-9e72a71efae4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:39.154477 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d05ddf3-71a5-4516-b208-9e72a71efae4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:39.155155 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d05ddf3-71a5-4516-b208-9e72a71efae4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:39.155484 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a451153-8150-4e9a-8d01-7efb674857d8 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:39.156766 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d05ddf3-71a5-4516-b208-9e72a71efae4 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:39.158451 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:55:39.160327 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db3a74bf-9ed6-4069-8252-6b449244a14d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:39.160948 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3a74bf-9ed6-4069-8252-6b449244a14d admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:39.161170 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3a74bf-9ed6-4069-8252-6b449244a14d admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:39.161468 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3a74bf-9ed6-4069-8252-6b449244a14d admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:39.162105 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db3a74bf-9ed6-4069-8252-6b449244a14d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:39.163673 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db3a74bf-9ed6-4069-8252-6b449244a14d admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:39.168918 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 802/1597] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:55:39 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:55:39.169897 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d05ddf3-71a5-4516-b208-9e72a71efae4 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:39.177355 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 796/1598] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:55:39 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:55:39.178518 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db3a74bf-9ed6-4069-8252-6b449244a14d admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:39.183009 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d05ddf3-71a5-4516-b208-9e72a71efae4 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:39.188333 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7baffdc-f978-463a-88ca-6b2fc0092101 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:39.188867 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db3a74bf-9ed6-4069-8252-6b449244a14d admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:39.189003 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7baffdc-f978-463a-88ca-6b2fc0092101 admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:39.189237 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7baffdc-f978-463a-88ca-6b2fc0092101 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:39.189543 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7baffdc-f978-463a-88ca-6b2fc0092101 admin admin] PATH_INFO: `/v3/projects/f09655506e354c9d991f4df9a92f4086/users/dbe551f8c10c4111b4ad7380a6482564/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:39.194460 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57594e59-125e-4f9f-be14-640863d44c75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:39.195100 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57594e59-125e-4f9f-be14-640863d44c75 admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:39.195379 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57594e59-125e-4f9f-be14-640863d44c75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:39.195649 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57594e59-125e-4f9f-be14-640863d44c75 admin admin] PATH_INFO: `/v3/projects/67751478ae3a45f89000e9f561524d2c/users/4fad770ae27641b3beb2be9f9c4673b6/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:39.205182 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7baffdc-f978-463a-88ca-6b2fc0092101 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f09655506e354c9d991f4df9a92f4086, user_id=dbe551f8c10c4111b4ad7380a6482564, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:39.207853 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7baffdc-f978-463a-88ca-6b2fc0092101 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:39.211390 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57594e59-125e-4f9f-be14-640863d44c75 admin admin] RBAC: Authorizing `identity:create_grant(project_id=67751478ae3a45f89000e9f561524d2c, user_id=4fad770ae27641b3beb2be9f9c4673b6, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:39.215322 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57594e59-125e-4f9f-be14-640863d44c75 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:39.223790 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 803/1599] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:55:39 2022] PUT /identity/v3/projects/f09655506e354c9d991f4df9a92f4086/users/dbe551f8c10c4111b4ad7380a6482564/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:55:39.224749 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7baffdc-f978-463a-88ca-6b2fc0092101 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:39.230104 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 797/1600] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:55:39 2022] PUT /identity/v3/projects/67751478ae3a45f89000e9f561524d2c/users/4fad770ae27641b3beb2be9f9c4673b6/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:55:39.231537 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57594e59-125e-4f9f-be14-640863d44c75 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:39.242104 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:55:39.295323 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57594e59-125e-4f9f-be14-640863d44c75 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:39.297207 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7baffdc-f978-463a-88ca-6b2fc0092101 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:39.300099 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10303e3-428c-41a2-965e-ad73ad4c7c56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:39.300890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10303e3-428c-41a2-965e-ad73ad4c7c56 admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:39.301159 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10303e3-428c-41a2-965e-ad73ad4c7c56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:39.301477 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10303e3-428c-41a2-965e-ad73ad4c7c56 admin admin] PATH_INFO: `/v3/projects/28568584195d475fbc52d76185e09044/users/8eabe2a74a604c47bc2e5c88b960dcb6/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:39.301835 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ee6699-d0e5-48d8-b3e9-471dd66f7537 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:39.302430 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ee6699-d0e5-48d8-b3e9-471dd66f7537 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:39.302712 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ee6699-d0e5-48d8-b3e9-471dd66f7537 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:39.302982 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ee6699-d0e5-48d8-b3e9-471dd66f7537 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:39.303735 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75ee6699-d0e5-48d8-b3e9-471dd66f7537 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:39.305214 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75ee6699-d0e5-48d8-b3e9-471dd66f7537 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:39.316168 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 804/1601] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:55:39 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:55:39.317186 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ee6699-d0e5-48d8-b3e9-471dd66f7537 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:39.320648 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d10303e3-428c-41a2-965e-ad73ad4c7c56 admin admin] RBAC: Authorizing `identity:create_grant(project_id=28568584195d475fbc52d76185e09044, user_id=8eabe2a74a604c47bc2e5c88b960dcb6, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:39.324457 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d10303e3-428c-41a2-965e-ad73ad4c7c56 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:39.338370 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 798/1602] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:55:39 2022] PUT /identity/v3/projects/28568584195d475fbc52d76185e09044/users/8eabe2a74a604c47bc2e5c88b960dcb6/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:55:39.339185 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10303e3-428c-41a2-965e-ad73ad4c7c56 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:39.396612 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10303e3-428c-41a2-965e-ad73ad4c7c56 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:39.396812 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ee6699-d0e5-48d8-b3e9-471dd66f7537 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:39.400718 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b305c0-c6b6-44a2-8c89-e3ad3983f61b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:39.401436 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b305c0-c6b6-44a2-8c89-e3ad3983f61b service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:39.401622 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b305c0-c6b6-44a2-8c89-e3ad3983f61b service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:39.401887 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b305c0-c6b6-44a2-8c89-e3ad3983f61b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:39.455731 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:55:39.464948 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a4f47d-1132-4cf3-8f64-23d71badf80f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f142988a4dd14a5f96eacea45c046f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:39.465593 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a4f47d-1132-4cf3-8f64-23d71badf80f service nova] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:39.465866 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a4f47d-1132-4cf3-8f64-23d71badf80f service nova] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:39.466131 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a4f47d-1132-4cf3-8f64-23d71badf80f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:39.466437 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5b305c0-c6b6-44a2-8c89-e3ad3983f61b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:39.467858 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5b305c0-c6b6-44a2-8c89-e3ad3983f61b service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:39.478981 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 799/1603] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:55:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:55:39.480186 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b305c0-c6b6-44a2-8c89-e3ad3983f61b service neutron] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:39.491561 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b305c0-c6b6-44a2-8c89-e3ad3983f61b service neutron] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:39.496731 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35aecf24-8a8c-4023-80c4-ccebce50c1cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:39.497433 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35aecf24-8a8c-4023-80c4-ccebce50c1cc admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:39.497725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35aecf24-8a8c-4023-80c4-ccebce50c1cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:39.497978 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35aecf24-8a8c-4023-80c4-ccebce50c1cc admin admin] PATH_INFO: `/v3/projects/5ca92ea462d24d2d8632e5c52a3dfe63/users/f6e6706abaea4d0797af19fdb0331ead/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:39.515943 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35aecf24-8a8c-4023-80c4-ccebce50c1cc admin admin] RBAC: Authorizing `identity:create_grant(project_id=5ca92ea462d24d2d8632e5c52a3dfe63, user_id=f6e6706abaea4d0797af19fdb0331ead, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:39.518961 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35aecf24-8a8c-4023-80c4-ccebce50c1cc admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:39.531313 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1a4f47d-1132-4cf3-8f64-23d71badf80f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:39.533198 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1a4f47d-1132-4cf3-8f64-23d71badf80f service nova] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:39.539049 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 800/1604] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:55:39 2022] PUT /identity/v3/projects/5ca92ea462d24d2d8632e5c52a3dfe63/users/f6e6706abaea4d0797af19fdb0331ead/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:55:39.539049 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35aecf24-8a8c-4023-80c4-ccebce50c1cc admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:39.547442 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 805/1605] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:55:39 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 231 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:55:39.550503 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342cd216-94a3-4c53-92fa-e2008b577394 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:39.550893 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342cd216-94a3-4c53-92fa-e2008b577394 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:39.551262 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342cd216-94a3-4c53-92fa-e2008b577394 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:39.563114 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:55:39.575506 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-342cd216-94a3-4c53-92fa-e2008b577394 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:55:39.577078 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-342cd216-94a3-4c53-92fa-e2008b577394 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:39.626899 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35aecf24-8a8c-4023-80c4-ccebce50c1cc admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:39.631875 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d1ef86-b6a2-4b17-91a8-ee3da91cd91d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:39.632685 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d1ef86-b6a2-4b17-91a8-ee3da91cd91d admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:39.633128 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d1ef86-b6a2-4b17-91a8-ee3da91cd91d admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:39.633599 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d1ef86-b6a2-4b17-91a8-ee3da91cd91d admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:39.635145 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85d1ef86-b6a2-4b17-91a8-ee3da91cd91d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:39.637052 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85d1ef86-b6a2-4b17-91a8-ee3da91cd91d admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:39.648544 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 801/1606] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:55:39 2022] GET /identity/v3/roles => generated 1420 bytes in 112 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:55:39.650625 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8a9b8a-0c4e-4a9c-9eaa-c2b525b3bbdd None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:39.650860 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8a9b8a-0c4e-4a9c-9eaa-c2b525b3bbdd None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:39.651310 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8a9b8a-0c4e-4a9c-9eaa-c2b525b3bbdd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:39.669141 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 806/1607] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:55:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:55:39.669867 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-342cd216-94a3-4c53-92fa-e2008b577394 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:39.683514 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-342cd216-94a3-4c53-92fa-e2008b577394 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:39.687499 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e6b459f-cf4f-45fb-9fa9-cbb8af706b80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:39.688115 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6b459f-cf4f-45fb-9fa9-cbb8af706b80 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:39.688435 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6b459f-cf4f-45fb-9fa9-cbb8af706b80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:39.689195 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6b459f-cf4f-45fb-9fa9-cbb8af706b80 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:39.689928 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e6b459f-cf4f-45fb-9fa9-cbb8af706b80 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:39.691519 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e6b459f-cf4f-45fb-9fa9-cbb8af706b80 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:39.696857 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-2e8a9b8a-0c4e-4a9c-9eaa-c2b525b3bbdd None None] MFA Rules not processed for user `4fad770ae27641b3beb2be9f9c4673b6`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:55:39.699418 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-2e8a9b8a-0c4e-4a9c-9eaa-c2b525b3bbdd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:39.704635 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 807/1608] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:55:39 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:55:39.705820 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce805f8-12b9-49b7-ab9a-a25c446d6877 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:39.706109 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce805f8-12b9-49b7-ab9a-a25c446d6877 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:39.706380 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce805f8-12b9-49b7-ab9a-a25c446d6877 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:39.730741 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-0ce805f8-12b9-49b7-ab9a-a25c446d6877 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:55:39.732018 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-0ce805f8-12b9-49b7-ab9a-a25c446d6877 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:39.741511 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 808/1609] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:55:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:55:39.742674 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12fb896-ad20-45b4-9281-351213c36107 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:39.742952 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12fb896-ad20-45b4-9281-351213c36107 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:39.743243 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12fb896-ad20-45b4-9281-351213c36107 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:39.766044 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-a12fb896-ad20-45b4-9281-351213c36107 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:55:39.767520 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-a12fb896-ad20-45b4-9281-351213c36107 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:39.779741 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 809/1610] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:55:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:55:39.779741 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a12fb896-ad20-45b4-9281-351213c36107 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:39.788626 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a12fb896-ad20-45b4-9281-351213c36107 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:39.794240 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74a7d5e8-cb73-4900-b105-bf257bfa886b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:39.794899 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a7d5e8-cb73-4900-b105-bf257bfa886b admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:39.795174 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a7d5e8-cb73-4900-b105-bf257bfa886b admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:39.795444 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a7d5e8-cb73-4900-b105-bf257bfa886b admin admin] PATH_INFO: `/v3/projects/8f6ac5d4eff54d17a8a5c5e39849f66a/users/90ccd5db1bfd42c3bdc2dbecf1ed0269/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:39.812039 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 802/1611] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:55:39 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:55:39.813233 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e8a9b8a-0c4e-4a9c-9eaa-c2b525b3bbdd None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:39.814617 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-2e8a9b8a-0c4e-4a9c-9eaa-c2b525b3bbdd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:39.818051 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74a7d5e8-cb73-4900-b105-bf257bfa886b admin admin] RBAC: Authorizing `identity:create_grant(project_id=8f6ac5d4eff54d17a8a5c5e39849f66a, user_id=90ccd5db1bfd42c3bdc2dbecf1ed0269, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:39.820925 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74a7d5e8-cb73-4900-b105-bf257bfa886b admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:39.841315 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 810/1612] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:55:39 2022] PUT /identity/v3/projects/8f6ac5d4eff54d17a8a5c5e39849f66a/users/90ccd5db1bfd42c3bdc2dbecf1ed0269/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:55:39.842240 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74a7d5e8-cb73-4900-b105-bf257bfa886b admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:39.894878 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e8a9b8a-0c4e-4a9c-9eaa-c2b525b3bbdd None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:39.925157 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:55:39.935357 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74a7d5e8-cb73-4900-b105-bf257bfa886b admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:39.942496 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f2dbb8-bc59-46b2-baaf-700bc9eeb462 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:39.943318 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f2dbb8-bc59-46b2-baaf-700bc9eeb462 admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:39.943593 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f2dbb8-bc59-46b2-baaf-700bc9eeb462 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:39.944004 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f2dbb8-bc59-46b2-baaf-700bc9eeb462 admin admin] PATH_INFO: `/v3/projects/412b366eb919434fa673464d0688b498/users/d579d0b72d984e7ca92324e3f5158cc0/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:39.962528 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d432c856-638e-48cc-96b9-bc459f2bdba2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:39.963170 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d432c856-638e-48cc-96b9-bc459f2bdba2 service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:39.963453 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d432c856-638e-48cc-96b9-bc459f2bdba2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:39.963722 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d432c856-638e-48cc-96b9-bc459f2bdba2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:39.965495 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1f2dbb8-bc59-46b2-baaf-700bc9eeb462 admin admin] RBAC: Authorizing `identity:create_grant(project_id=412b366eb919434fa673464d0688b498, user_id=d579d0b72d984e7ca92324e3f5158cc0, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:39.968162 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1f2dbb8-bc59-46b2-baaf-700bc9eeb462 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:39.974136 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d432c856-638e-48cc-96b9-bc459f2bdba2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:39.975570 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d432c856-638e-48cc-96b9-bc459f2bdba2 service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:39.985713 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 811/1613] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:55:39 2022] PUT /identity/v3/projects/412b366eb919434fa673464d0688b498/users/d579d0b72d984e7ca92324e3f5158cc0/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:55:39.986847 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f2dbb8-bc59-46b2-baaf-700bc9eeb462 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:39.987338 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 803/1614] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:55:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:55:39.988480 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d432c856-638e-48cc-96b9-bc459f2bdba2 service neutron] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:39.993315 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:55:40.000768 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d432c856-638e-48cc-96b9-bc459f2bdba2 service neutron] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:40.066451 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ef1607-1142-4c36-9892-530f0830e72f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:40.066852 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ef1607-1142-4c36-9892-530f0830e72f service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:40.067132 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ef1607-1142-4c36-9892-530f0830e72f service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:40.067536 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ef1607-1142-4c36-9892-530f0830e72f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:40.072777 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f2dbb8-bc59-46b2-baaf-700bc9eeb462 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:40.077249 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-558a2340-3f99-4e98-b208-e5de3e65d615 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:40.078075 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558a2340-3f99-4e98-b208-e5de3e65d615 service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:40.078479 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558a2340-3f99-4e98-b208-e5de3e65d615 service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:40.078632 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558a2340-3f99-4e98-b208-e5de3e65d615 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:40.142764 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4ef1607-1142-4c36-9892-530f0830e72f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:40.144313 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4ef1607-1142-4c36-9892-530f0830e72f service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:40.152721 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-558a2340-3f99-4e98-b208-e5de3e65d615 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:40.154566 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-558a2340-3f99-4e98-b208-e5de3e65d615 service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:40.155917 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 804/1615] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:55:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:55:40.156698 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ef1607-1142-4c36-9892-530f0830e72f service neutron] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:40.166487 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 812/1616] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:55:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:55:40.167878 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54fdd379-04a2-4114-a000-a1f10d5a35f7 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:40.168804 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54fdd379-04a2-4114-a000-a1f10d5a35f7 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:40.168804 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54fdd379-04a2-4114-a000-a1f10d5a35f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:40.203610 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-54fdd379-04a2-4114-a000-a1f10d5a35f7 None None] MFA Rules not processed for user `90ccd5db1bfd42c3bdc2dbecf1ed0269`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:55:40.205076 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-54fdd379-04a2-4114-a000-a1f10d5a35f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:40.223687 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ef1607-1142-4c36-9892-530f0830e72f service neutron] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:40.228488 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d758352-bd1e-4317-9420-7343161d0418 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:40.228705 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d758352-bd1e-4317-9420-7343161d0418 service cinder] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:40.229120 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d758352-bd1e-4317-9420-7343161d0418 service cinder] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:40.229504 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d758352-bd1e-4317-9420-7343161d0418 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:40.287438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 813/1617] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:55:40 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:55:40.289555 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf95fba-c867-4328-9b24-baacec10b85e None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:40.290050 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf95fba-c867-4328-9b24-baacec10b85e None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:40.291068 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf95fba-c867-4328-9b24-baacec10b85e None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:40.294101 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 814/1618] 10.222.0.52 () {54 vars in 912 bytes} [Sun Sep 25 00:55:40 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:55:40.295704 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a408ad0-8101-46cc-965d-cf6f6a1860d1 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:40.295997 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a408ad0-8101-46cc-965d-cf6f6a1860d1 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:40.296405 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a408ad0-8101-46cc-965d-cf6f6a1860d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:40.307749 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d758352-bd1e-4317-9420-7343161d0418 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:40.309350 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d758352-bd1e-4317-9420-7343161d0418 service cinder] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:40.321073 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 805/1619] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:55:40 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 165 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:55:40.322346 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d758352-bd1e-4317-9420-7343161d0418 service cinder] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:40.334276 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d758352-bd1e-4317-9420-7343161d0418 service cinder] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:40.340678 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b6cbf54-5be6-4e76-bf00-adf2ffab47a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:40.340900 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-8a408ad0-8101-46cc-965d-cf6f6a1860d1 None None] MFA Rules not processed for user `d579d0b72d984e7ca92324e3f5158cc0`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:55:40.341373 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6cbf54-5be6-4e76-bf00-adf2ffab47a2 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:40.341664 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6cbf54-5be6-4e76-bf00-adf2ffab47a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:40.341937 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6cbf54-5be6-4e76-bf00-adf2ffab47a2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:40.342120 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-8a408ad0-8101-46cc-965d-cf6f6a1860d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:40.342729 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b6cbf54-5be6-4e76-bf00-adf2ffab47a2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:40.344644 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b6cbf54-5be6-4e76-bf00-adf2ffab47a2 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:40.351006 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:55:40.351205 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:55:40.392930 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 806/1620] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:55:40 2022] POST /identity/v3/projects => generated 405 bytes in 68 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:55:40.392930 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3e979d-f477-41e7-8179-1b46b1e58a57 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:40.392930 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3e979d-f477-41e7-8179-1b46b1e58a57 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:40.392930 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3e979d-f477-41e7-8179-1b46b1e58a57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:40.416717 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-9b3e979d-f477-41e7-8179-1b46b1e58a57 None None] MFA Rules not processed for user `f142988a4dd14a5f96eacea45c046f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:55:40.418169 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-9b3e979d-f477-41e7-8179-1b46b1e58a57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:40.443213 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 815/1621] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:55:40 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:55:40.444058 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a408ad0-8101-46cc-965d-cf6f6a1860d1 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:40.464857 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a408ad0-8101-46cc-965d-cf6f6a1860d1 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:40.506243 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 807/1622] 10.222.0.52 () {58 vars in 1009 bytes} [Sun Sep 25 00:55:40 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:55:40.507084 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3e979d-f477-41e7-8179-1b46b1e58a57 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:40.539349 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d9ec0b0-577a-4f70-a3f4-37edfae6d9ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:40.541603 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9ec0b0-577a-4f70-a3f4-37edfae6d9ea admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:40.541603 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9ec0b0-577a-4f70-a3f4-37edfae6d9ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:40.541930 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9ec0b0-577a-4f70-a3f4-37edfae6d9ea admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:40.542859 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d9ec0b0-577a-4f70-a3f4-37edfae6d9ea admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:40.546236 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d9ec0b0-577a-4f70-a3f4-37edfae6d9ea admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:40.583517 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 816/1623] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:55:40 2022] POST /identity/v3/users => generated 309 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:55:40.584692 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d9ec0b0-577a-4f70-a3f4-37edfae6d9ea admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:40.595220 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3e979d-f477-41e7-8179-1b46b1e58a57 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:40.600677 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0796046-1e9b-4c0c-9753-51d7c3603239 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:40.601408 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0796046-1e9b-4c0c-9753-51d7c3603239 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:40.601784 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0796046-1e9b-4c0c-9753-51d7c3603239 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:40.602066 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0796046-1e9b-4c0c-9753-51d7c3603239 admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:40.603467 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0796046-1e9b-4c0c-9753-51d7c3603239 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:40.606433 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0796046-1e9b-4c0c-9753-51d7c3603239 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:40.666269 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 808/1624] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:55:40 2022] POST /identity/v3/projects => generated 371 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:55:40.667218 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0796046-1e9b-4c0c-9753-51d7c3603239 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:40.679348 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d9ec0b0-577a-4f70-a3f4-37edfae6d9ea admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:40.758399 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3e8968d-2a43-4da2-8a67-dd612907911b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:40.759688 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e8968d-2a43-4da2-8a67-dd612907911b service cinder] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:40.760072 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e8968d-2a43-4da2-8a67-dd612907911b service cinder] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:40.760489 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e8968d-2a43-4da2-8a67-dd612907911b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:40.761143 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0796046-1e9b-4c0c-9753-51d7c3603239 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:40.766422 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:55:40.768006 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-786ce775-067c-4509-b594-902de4eed00d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:40.770167 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786ce775-067c-4509-b594-902de4eed00d admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:40.770292 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786ce775-067c-4509-b594-902de4eed00d admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:40.770492 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786ce775-067c-4509-b594-902de4eed00d admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:40.771366 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-786ce775-067c-4509-b594-902de4eed00d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:40.773085 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-786ce775-067c-4509-b594-902de4eed00d admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:40.776848 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:55:40.787353 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 809/1625] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:55:40 2022] GET /identity/v3/roles => generated 1420 bytes in 121 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:55:40.788945 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-786ce775-067c-4509-b594-902de4eed00d admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:40.802868 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-786ce775-067c-4509-b594-902de4eed00d admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:40.809746 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ff9afb-d1a1-4c19-85d9-3e92ae46ee6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:40.810714 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ff9afb-d1a1-4c19-85d9-3e92ae46ee6b admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:40.811151 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ff9afb-d1a1-4c19-85d9-3e92ae46ee6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:40.811632 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ff9afb-d1a1-4c19-85d9-3e92ae46ee6b admin admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:40.812775 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ff9afb-d1a1-4c19-85d9-3e92ae46ee6b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:40.815659 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ff9afb-d1a1-4c19-85d9-3e92ae46ee6b admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:40.842872 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 810/1626] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:55:40 2022] POST /identity/v3/users => generated 292 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:55:40.843713 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ff9afb-d1a1-4c19-85d9-3e92ae46ee6b admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:40.856784 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ff9afb-d1a1-4c19-85d9-3e92ae46ee6b admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:40.864821 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366d8620-8a99-4bd7-99a0-57677f598405 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:40.865500 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366d8620-8a99-4bd7-99a0-57677f598405 admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:40.865770 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366d8620-8a99-4bd7-99a0-57677f598405 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:40.866047 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366d8620-8a99-4bd7-99a0-57677f598405 admin admin] PATH_INFO: `/v3/projects/966cbe725c7a45dda78e7ac72e56a7b1/users/a669fd0abc5c4554922f2693a217a301/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:40.886184 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-366d8620-8a99-4bd7-99a0-57677f598405 admin admin] RBAC: Authorizing `identity:create_grant(project_id=966cbe725c7a45dda78e7ac72e56a7b1, user_id=a669fd0abc5c4554922f2693a217a301, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:40.890383 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-366d8620-8a99-4bd7-99a0-57677f598405 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:40.915711 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 811/1627] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:55:40 2022] PUT /identity/v3/projects/966cbe725c7a45dda78e7ac72e56a7b1/users/a669fd0abc5c4554922f2693a217a301/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 71 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:55:40.918477 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366d8620-8a99-4bd7-99a0-57677f598405 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:40.921933 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3e8968d-2a43-4da2-8a67-dd612907911b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:40.927929 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3e8968d-2a43-4da2-8a67-dd612907911b service cinder] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:40.955804 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 817/1628] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:55:40 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:55:40.960451 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3e8968d-2a43-4da2-8a67-dd612907911b service cinder] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:40.991089 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3e8968d-2a43-4da2-8a67-dd612907911b service cinder] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:41.093511 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366d8620-8a99-4bd7-99a0-57677f598405 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:41.110494 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ad6378-286e-4243-a7bf-1797eb04ade7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:41.111129 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ad6378-286e-4243-a7bf-1797eb04ade7 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:41.111409 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ad6378-286e-4243-a7bf-1797eb04ade7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:41.111680 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ad6378-286e-4243-a7bf-1797eb04ade7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:41.112329 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7ad6378-286e-4243-a7bf-1797eb04ade7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:41.114231 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7ad6378-286e-4243-a7bf-1797eb04ade7 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:41.138492 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 812/1629] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:55:40 2022] GET /identity/v3/roles => generated 1420 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:55:41.149410 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ad6378-286e-4243-a7bf-1797eb04ade7 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:41.157447 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933dea10-62b3-4be1-9067-1f43bd65a526 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:41.158630 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933dea10-62b3-4be1-9067-1f43bd65a526 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:41.161725 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933dea10-62b3-4be1-9067-1f43bd65a526 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:41.162264 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933dea10-62b3-4be1-9067-1f43bd65a526 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:41.163153 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-933dea10-62b3-4be1-9067-1f43bd65a526 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:41.166378 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-933dea10-62b3-4be1-9067-1f43bd65a526 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:41.179119 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ad6378-286e-4243-a7bf-1797eb04ade7 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:41.191385 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d7280cf-60a6-4995-a52e-03bada6a707a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:41.192433 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7280cf-60a6-4995-a52e-03bada6a707a admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:41.192718 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7280cf-60a6-4995-a52e-03bada6a707a admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:41.194371 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7280cf-60a6-4995-a52e-03bada6a707a admin admin] PATH_INFO: `/v3/projects/cc00f38ca5bb4d34806d54bc8967f23c/users/c783d9a1a37c47b1bb635d2bed48f3b1/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:41.211678 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 818/1630] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:55:40 2022] GET /identity/v3/roles => generated 1420 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:55:41.217346 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933dea10-62b3-4be1-9067-1f43bd65a526 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:41.242525 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d7280cf-60a6-4995-a52e-03bada6a707a admin admin] RBAC: Authorizing `identity:create_grant(project_id=cc00f38ca5bb4d34806d54bc8967f23c, user_id=c783d9a1a37c47b1bb635d2bed48f3b1, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:41.251065 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d7280cf-60a6-4995-a52e-03bada6a707a admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:41.257446 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933dea10-62b3-4be1-9067-1f43bd65a526 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:41.266971 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29696c4f-2aa9-4e30-b59f-c961b3eb7fdf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:41.266971 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29696c4f-2aa9-4e30-b59f-c961b3eb7fdf admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:41.266971 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29696c4f-2aa9-4e30-b59f-c961b3eb7fdf admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:41.266971 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29696c4f-2aa9-4e30-b59f-c961b3eb7fdf admin admin] PATH_INFO: `/v3/projects/966cbe725c7a45dda78e7ac72e56a7b1/users/a669fd0abc5c4554922f2693a217a301/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:41.269345 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29696c4f-2aa9-4e30-b59f-c961b3eb7fdf admin admin] RBAC: Authorizing `identity:create_grant(project_id=966cbe725c7a45dda78e7ac72e56a7b1, user_id=a669fd0abc5c4554922f2693a217a301, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:41.272645 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29696c4f-2aa9-4e30-b59f-c961b3eb7fdf admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:41.289463 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 813/1631] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:55:41 2022] PUT /identity/v3/projects/cc00f38ca5bb4d34806d54bc8967f23c/users/c783d9a1a37c47b1bb635d2bed48f3b1/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 00:55:41.291741 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 819/1632] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:55:41 2022] PUT /identity/v3/projects/966cbe725c7a45dda78e7ac72e56a7b1/users/a669fd0abc5c4554922f2693a217a301/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 75 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:55:41.295080 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d7280cf-60a6-4995-a52e-03bada6a707a admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:41.306878 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09222eb6-8cf2-4aea-a249-8d5a3e677aeb None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:41.307304 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09222eb6-8cf2-4aea-a249-8d5a3e677aeb None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:41.308369 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09222eb6-8cf2-4aea-a249-8d5a3e677aeb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:41.376231 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-09222eb6-8cf2-4aea-a249-8d5a3e677aeb None None] MFA Rules not processed for user `a669fd0abc5c4554922f2693a217a301`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:55:41.380452 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-09222eb6-8cf2-4aea-a249-8d5a3e677aeb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:41.456187 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d7280cf-60a6-4995-a52e-03bada6a707a admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:41.529345 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:55:41.554129 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 820/1633] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:55:41 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 251 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:55:41.564343 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c02666-f04a-4e09-a4c8-f7f67dbdb7bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:41.565100 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c02666-f04a-4e09-a4c8-f7f67dbdb7bc admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:41.565502 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c02666-f04a-4e09-a4c8-f7f67dbdb7bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:41.565863 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c02666-f04a-4e09-a4c8-f7f67dbdb7bc admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:41.566399 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09222eb6-8cf2-4aea-a249-8d5a3e677aeb None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:41.566739 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2c02666-f04a-4e09-a4c8-f7f67dbdb7bc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:41.568873 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2c02666-f04a-4e09-a4c8-f7f67dbdb7bc admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:41.572841 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:55:41.605466 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 814/1634] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:55:41 2022] GET /identity/v3/roles => generated 1420 bytes in 311 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:55:41.607205 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c02666-f04a-4e09-a4c8-f7f67dbdb7bc admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:41.624369 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c02666-f04a-4e09-a4c8-f7f67dbdb7bc admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:41.635856 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fceaf0f9-8d8d-4a4d-8f3a-3da8f4072272 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:41.636987 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fceaf0f9-8d8d-4a4d-8f3a-3da8f4072272 admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:41.637466 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fceaf0f9-8d8d-4a4d-8f3a-3da8f4072272 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:41.638133 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fceaf0f9-8d8d-4a4d-8f3a-3da8f4072272 admin admin] PATH_INFO: `/v3/projects/cc00f38ca5bb4d34806d54bc8967f23c/users/c783d9a1a37c47b1bb635d2bed48f3b1/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:41.642562 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fceaf0f9-8d8d-4a4d-8f3a-3da8f4072272 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cc00f38ca5bb4d34806d54bc8967f23c, user_id=c783d9a1a37c47b1bb635d2bed48f3b1, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:41.647032 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fceaf0f9-8d8d-4a4d-8f3a-3da8f4072272 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:41.676798 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 815/1635] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:55:41 2022] PUT /identity/v3/projects/cc00f38ca5bb4d34806d54bc8967f23c/users/c783d9a1a37c47b1bb635d2bed48f3b1/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:55:41.680479 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b9df2c-fbfb-4fa0-96d3-ccbff1901430 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:41.681355 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b9df2c-fbfb-4fa0-96d3-ccbff1901430 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:41.681783 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b9df2c-fbfb-4fa0-96d3-ccbff1901430 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:41.683776 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09222eb6-8cf2-4aea-a249-8d5a3e677aeb None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:41.726024 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-51b9df2c-fbfb-4fa0-96d3-ccbff1901430 None None] MFA Rules not processed for user `c783d9a1a37c47b1bb635d2bed48f3b1`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:55:41.727643 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-51b9df2c-fbfb-4fa0-96d3-ccbff1901430 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:41.808000 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-657a1215-9337-4f9c-8e83-95d7a1107a1a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:41.808625 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657a1215-9337-4f9c-8e83-95d7a1107a1a service cinder] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:41.808896 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657a1215-9337-4f9c-8e83-95d7a1107a1a service cinder] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:41.809160 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657a1215-9337-4f9c-8e83-95d7a1107a1a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:41.904638 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 816/1636] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:55:41 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 227 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:55:41.925375 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b9df2c-fbfb-4fa0-96d3-ccbff1901430 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:41.942684 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b9df2c-fbfb-4fa0-96d3-ccbff1901430 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:41.952612 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0fcd931-56a8-4a35-b95e-f26a660877e9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:41.953591 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0fcd931-56a8-4a35-b95e-f26a660877e9 service cinder] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:41.953868 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0fcd931-56a8-4a35-b95e-f26a660877e9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:41.954148 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0fcd931-56a8-4a35-b95e-f26a660877e9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:41.970315 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0fcd931-56a8-4a35-b95e-f26a660877e9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:41.976202 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0fcd931-56a8-4a35-b95e-f26a660877e9 service cinder] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:41.984279 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-657a1215-9337-4f9c-8e83-95d7a1107a1a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:41.985813 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-657a1215-9337-4f9c-8e83-95d7a1107a1a service cinder] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:42.000071 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 817/1637] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:55:41 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 76 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 00:55:42.033424 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 821/1638] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:55:41 2022] GET /identity/v3/auth/tokens => generated 3349 bytes in 468 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 00:55:42.602497 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0fcd931-56a8-4a35-b95e-f26a660877e9 service cinder] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:42.770037 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0fcd931-56a8-4a35-b95e-f26a660877e9 service cinder] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:42.783843 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a7eebfa-cd40-47c7-9381-dfa61d44ce4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:42.784500 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7eebfa-cd40-47c7-9381-dfa61d44ce4b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:42.784783 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7eebfa-cd40-47c7-9381-dfa61d44ce4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:42.787288 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7eebfa-cd40-47c7-9381-dfa61d44ce4b admin admin] PATH_INFO: `/v3/users/90ccd5db1bfd42c3bdc2dbecf1ed0269` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:42.788975 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a7eebfa-cd40-47c7-9381-dfa61d44ce4b admin admin] RBAC: Authorizing `identity:delete_user(user_id=90ccd5db1bfd42c3bdc2dbecf1ed0269)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:42.793055 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b607cd-2583-49cd-84af-34accb4861a1 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:42.794426 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b607cd-2583-49cd-84af-34accb4861a1 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:42.794898 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b607cd-2583-49cd-84af-34accb4861a1 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:42.796635 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 822/1639] 10.222.0.52 () {54 vars in 961 bytes} [Sun Sep 25 00:55:42 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 00:55:42.797861 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a7eebfa-cd40-47c7-9381-dfa61d44ce4b admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:42.817145 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7798d1f-6512-4869-8531-546d3db5c347 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:42.817880 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7798d1f-6512-4869-8531-546d3db5c347 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:42.818429 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7798d1f-6512-4869-8531-546d3db5c347 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:42.885745 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-f7798d1f-6512-4869-8531-546d3db5c347 None None] MFA Rules not processed for user `f142988a4dd14a5f96eacea45c046f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:55:42.887578 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=94,keystone:INSERT=1,keystone:DELETE=5 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:55:42.888424 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-f7798d1f-6512-4869-8531-546d3db5c347 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:42.928678 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-2a7eebfa-cd40-47c7-9381-dfa61d44ce4b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '90ccd5db1bfd42c3bdc2dbecf1ed0269'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:42.929516 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-2a7eebfa-cd40-47c7-9381-dfa61d44ce4b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '90ccd5db1bfd42c3bdc2dbecf1ed0269'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:42.942814 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-2a7eebfa-cd40-47c7-9381-dfa61d44ce4b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '90ccd5db1bfd42c3bdc2dbecf1ed0269'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:42.943393 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-2a7eebfa-cd40-47c7-9381-dfa61d44ce4b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '90ccd5db1bfd42c3bdc2dbecf1ed0269'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:42.973437 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-2a7eebfa-cd40-47c7-9381-dfa61d44ce4b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '90ccd5db1bfd42c3bdc2dbecf1ed0269'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:43.055091 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 818/1640] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:55:42 2022] DELETE /identity/v3/users/90ccd5db1bfd42c3bdc2dbecf1ed0269 => generated 0 bytes in 454 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:55:43.062818 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96795c4-93ed-4555-b27b-40e56d0e9ca6 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:43.062953 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:55:43.063125 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96795c4-93ed-4555-b27b-40e56d0e9ca6 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:43.063365 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96795c4-93ed-4555-b27b-40e56d0e9ca6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:43.107877 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 823/1641] 10.222.0.52 () {58 vars in 1058 bytes} [Sun Sep 25 00:55:42 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 292 msecs (HTTP/1.1 201) 6 headers in 385 bytes (3 switches on core 0) Sep 25 00:55:43.115712 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9180c601-3d8a-4df2-ada6-28bb04f1f426 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:43.115997 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9180c601-3d8a-4df2-ada6-28bb04f1f426 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:43.116268 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9180c601-3d8a-4df2-ada6-28bb04f1f426 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:43.124554 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-a96795c4-93ed-4555-b27b-40e56d0e9ca6 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:55:43.127679 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 824/1642] 10.222.0.52 () {54 vars in 912 bytes} [Sun Sep 25 00:55:43 2022] GET /identity => generated 270 bytes in 13 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:55:43.129237 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ba621f-706e-42e1-9061-9aac1634dbba None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:43.129550 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ba621f-706e-42e1-9061-9aac1634dbba None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:43.129821 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ba621f-706e-42e1-9061-9aac1634dbba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:43.131268 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-a96795c4-93ed-4555-b27b-40e56d0e9ca6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:43.192935 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-33ba621f-706e-42e1-9061-9aac1634dbba None None] MFA Rules not processed for user `f142988a4dd14a5f96eacea45c046f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:55:43.194474 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-33ba621f-706e-42e1-9061-9aac1634dbba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:43.344147 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 819/1643] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:55:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 283 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:55:43.346446 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96795c4-93ed-4555-b27b-40e56d0e9ca6 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:43.350413 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-a96795c4-93ed-4555-b27b-40e56d0e9ca6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:43.376141 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96795c4-93ed-4555-b27b-40e56d0e9ca6 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:43.379086 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 825/1644] 10.222.0.52 () {58 vars in 1009 bytes} [Sun Sep 25 00:55:43 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 251 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:55:43.383046 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e4952f-d52c-4297-9052-3c5dce0df602 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:43.384643 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e4952f-d52c-4297-9052-3c5dce0df602 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:43.385082 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e4952f-d52c-4297-9052-3c5dce0df602 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:43.385687 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e4952f-d52c-4297-9052-3c5dce0df602 admin admin] PATH_INFO: `/v3/users/d579d0b72d984e7ca92324e3f5158cc0` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:43.390652 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e4952f-d52c-4297-9052-3c5dce0df602 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d579d0b72d984e7ca92324e3f5158cc0)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:43.415886 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ba621f-706e-42e1-9061-9aac1634dbba None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:43.415886 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-33ba621f-706e-42e1-9061-9aac1634dbba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:43.416416 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e4952f-d52c-4297-9052-3c5dce0df602 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:43.523623 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ba621f-706e-42e1-9061-9aac1634dbba None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:43.526339 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-12e4952f-d52c-4297-9052-3c5dce0df602 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd579d0b72d984e7ca92324e3f5158cc0'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:43.527024 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-12e4952f-d52c-4297-9052-3c5dce0df602 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd579d0b72d984e7ca92324e3f5158cc0'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:43.533461 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d435220-0fe3-47df-a83d-580409b6be8a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:43.534074 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d435220-0fe3-47df-a83d-580409b6be8a service cinder] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:43.534322 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d435220-0fe3-47df-a83d-580409b6be8a service cinder] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:43.534594 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d435220-0fe3-47df-a83d-580409b6be8a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:43.544869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-12e4952f-d52c-4297-9052-3c5dce0df602 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd579d0b72d984e7ca92324e3f5158cc0'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:43.545593 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-12e4952f-d52c-4297-9052-3c5dce0df602 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd579d0b72d984e7ca92324e3f5158cc0'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:43.551638 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d435220-0fe3-47df-a83d-580409b6be8a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:43.553859 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d435220-0fe3-47df-a83d-580409b6be8a service cinder] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:43.558708 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-12e4952f-d52c-4297-9052-3c5dce0df602 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd579d0b72d984e7ca92324e3f5158cc0'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:43.591328 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 826/1645] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:55:43 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 196 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:55:43.592584 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d435220-0fe3-47df-a83d-580409b6be8a service cinder] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:43.593997 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-9d435220-0fe3-47df-a83d-580409b6be8a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:43.613132 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 820/1646] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:55:43 2022] DELETE /identity/v3/users/d579d0b72d984e7ca92324e3f5158cc0 => generated 0 bytes in 267 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:55:43.618265 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae67a92-a89f-44d3-bca1-da0e9c8ec70c None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:43.618974 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae67a92-a89f-44d3-bca1-da0e9c8ec70c None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:43.619434 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae67a92-a89f-44d3-bca1-da0e9c8ec70c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:43.666556 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-8ae67a92-a89f-44d3-bca1-da0e9c8ec70c None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:55:43.672315 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-8ae67a92-a89f-44d3-bca1-da0e9c8ec70c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:43.724774 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d435220-0fe3-47df-a83d-580409b6be8a service cinder] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:43.745485 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:55:43.874212 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 821/1647] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:55:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 254 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:55:43.895020 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b2dd82-2a1d-4e33-abb5-8b7016825eac service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:43.895709 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b2dd82-2a1d-4e33-abb5-8b7016825eac service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:43.896087 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b2dd82-2a1d-4e33-abb5-8b7016825eac service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:43.896458 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b2dd82-2a1d-4e33-abb5-8b7016825eac service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:43.900121 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-40b2dd82-2a1d-4e33-abb5-8b7016825eac service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:43.926694 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:55:43.934028 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b2dd82-2a1d-4e33-abb5-8b7016825eac service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:43.935301 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b2dd82-2a1d-4e33-abb5-8b7016825eac service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:43.956462 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 827/1648] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:55:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 364 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:55:43.979254 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae67a92-a89f-44d3-bca1-da0e9c8ec70c None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:43.981375 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-8ae67a92-a89f-44d3-bca1-da0e9c8ec70c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:44.003385 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae67a92-a89f-44d3-bca1-da0e9c8ec70c None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:44.016240 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8373d8c9-97ef-4df0-92b3-35920424eb58 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:44.017016 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8373d8c9-97ef-4df0-92b3-35920424eb58 service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:44.017421 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8373d8c9-97ef-4df0-92b3-35920424eb58 service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:44.017790 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8373d8c9-97ef-4df0-92b3-35920424eb58 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:44.043092 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8373d8c9-97ef-4df0-92b3-35920424eb58 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:44.044755 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8373d8c9-97ef-4df0-92b3-35920424eb58 service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:44.057387 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 822/1649] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:55:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:55:45.858763 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b2dd82-2a1d-4e33-abb5-8b7016825eac service neutron] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:45.860372 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-40b2dd82-2a1d-4e33-abb5-8b7016825eac service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:45.894870 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b2dd82-2a1d-4e33-abb5-8b7016825eac service neutron] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:45.912033 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc809d9-922d-4b1a-84ee-d3e77ebd29f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:45.912703 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc809d9-922d-4b1a-84ee-d3e77ebd29f5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:45.914058 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc809d9-922d-4b1a-84ee-d3e77ebd29f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:45.914331 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc809d9-922d-4b1a-84ee-d3e77ebd29f5 admin admin] PATH_INFO: `/v3/projects/412b366eb919434fa673464d0688b498` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:45.919646 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fc809d9-922d-4b1a-84ee-d3e77ebd29f5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=412b366eb919434fa673464d0688b498)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:45.922403 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fc809d9-922d-4b1a-84ee-d3e77ebd29f5 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:46.037143 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-1fc809d9-922d-4b1a-84ee-d3e77ebd29f5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '412b366eb919434fa673464d0688b498', 'request_id': 'req-1fc809d9-922d-4b1a-84ee-d3e77ebd29f5'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:46.057275 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-1fc809d9-922d-4b1a-84ee-d3e77ebd29f5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '412b366eb919434fa673464d0688b498', 'request_id': 'req-1fc809d9-922d-4b1a-84ee-d3e77ebd29f5'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:46.063115 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-1fc809d9-922d-4b1a-84ee-d3e77ebd29f5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '412b366eb919434fa673464d0688b498', 'request_id': 'req-1fc809d9-922d-4b1a-84ee-d3e77ebd29f5'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:46.096566 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-1fc809d9-922d-4b1a-84ee-d3e77ebd29f5 admin admin] The token cache is being invalidate because project 412b366eb919434fa673464d0688b498 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81386) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:55:46.096853 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-1fc809d9-922d-4b1a-84ee-d3e77ebd29f5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:46.097727 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-1fc809d9-922d-4b1a-84ee-d3e77ebd29f5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:46.099297 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 828/1650] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:55:45 2022] DELETE /identity/v3/projects/412b366eb919434fa673464d0688b498 => generated 0 bytes in 241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:55:46.104097 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8373d8c9-97ef-4df0-92b3-35920424eb58 service neutron] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:46.118635 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8373d8c9-97ef-4df0-92b3-35920424eb58 service neutron] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:46.254637 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd40157-c3ea-4fd2-834f-8fc9a3214802 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:46.255288 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd40157-c3ea-4fd2-834f-8fc9a3214802 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:46.255562 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd40157-c3ea-4fd2-834f-8fc9a3214802 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:46.255832 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd40157-c3ea-4fd2-834f-8fc9a3214802 admin admin] PATH_INFO: `/v3/users/c783d9a1a37c47b1bb635d2bed48f3b1` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:46.262420 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd40157-c3ea-4fd2-834f-8fc9a3214802 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c783d9a1a37c47b1bb635d2bed48f3b1)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:46.265203 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd40157-c3ea-4fd2-834f-8fc9a3214802 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:46.369144 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc809d9-922d-4b1a-84ee-d3e77ebd29f5 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:46.371357 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-1fc809d9-922d-4b1a-84ee-d3e77ebd29f5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:46.408568 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc809d9-922d-4b1a-84ee-d3e77ebd29f5 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:46.419231 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d92ee7-48b3-4e9f-92b4-051300506721 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:46.421963 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d92ee7-48b3-4e9f-92b4-051300506721 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:46.423299 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d92ee7-48b3-4e9f-92b4-051300506721 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:46.423745 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d92ee7-48b3-4e9f-92b4-051300506721 admin admin] PATH_INFO: `/v3/projects/8f6ac5d4eff54d17a8a5c5e39849f66a` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:46.429155 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2d92ee7-48b3-4e9f-92b4-051300506721 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8f6ac5d4eff54d17a8a5c5e39849f66a)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:46.429776 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-6cd40157-c3ea-4fd2-834f-8fc9a3214802 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c783d9a1a37c47b1bb635d2bed48f3b1'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:46.431709 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-6cd40157-c3ea-4fd2-834f-8fc9a3214802 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c783d9a1a37c47b1bb635d2bed48f3b1'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:46.436888 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2d92ee7-48b3-4e9f-92b4-051300506721 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:46.448709 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-6cd40157-c3ea-4fd2-834f-8fc9a3214802 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c783d9a1a37c47b1bb635d2bed48f3b1'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:46.449733 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-6cd40157-c3ea-4fd2-834f-8fc9a3214802 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c783d9a1a37c47b1bb635d2bed48f3b1'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:46.473803 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-6cd40157-c3ea-4fd2-834f-8fc9a3214802 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c783d9a1a37c47b1bb635d2bed48f3b1'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:46.539471 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 823/1651] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:55:46 2022] DELETE /identity/v3/users/c783d9a1a37c47b1bb635d2bed48f3b1 => generated 0 bytes in 433 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:55:46.584297 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-d2d92ee7-48b3-4e9f-92b4-051300506721 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8f6ac5d4eff54d17a8a5c5e39849f66a', 'request_id': 'req-d2d92ee7-48b3-4e9f-92b4-051300506721'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:46.602009 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-d2d92ee7-48b3-4e9f-92b4-051300506721 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8f6ac5d4eff54d17a8a5c5e39849f66a', 'request_id': 'req-d2d92ee7-48b3-4e9f-92b4-051300506721'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:46.623064 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-d2d92ee7-48b3-4e9f-92b4-051300506721 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8f6ac5d4eff54d17a8a5c5e39849f66a', 'request_id': 'req-d2d92ee7-48b3-4e9f-92b4-051300506721'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:46.669047 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-d2d92ee7-48b3-4e9f-92b4-051300506721 admin admin] The token cache is being invalidate because project 8f6ac5d4eff54d17a8a5c5e39849f66a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81386) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:55:46.669594 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-d2d92ee7-48b3-4e9f-92b4-051300506721 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:46.671779 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-d2d92ee7-48b3-4e9f-92b4-051300506721 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:46.674555 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 829/1652] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:55:46 2022] DELETE /identity/v3/projects/8f6ac5d4eff54d17a8a5c5e39849f66a => generated 0 bytes in 306 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:55:46.678805 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd40157-c3ea-4fd2-834f-8fc9a3214802 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:46.681045 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-6cd40157-c3ea-4fd2-834f-8fc9a3214802 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:46.760048 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:55:46.802467 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd40157-c3ea-4fd2-834f-8fc9a3214802 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:46.937853 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b040b4-abe0-4576-859c-1e8243534e88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:46.938528 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b040b4-abe0-4576-859c-1e8243534e88 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:46.938807 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b040b4-abe0-4576-859c-1e8243534e88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:46.939075 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b040b4-abe0-4576-859c-1e8243534e88 admin admin] PATH_INFO: `/v3/users/a669fd0abc5c4554922f2693a217a301` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:46.940337 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b040b4-abe0-4576-859c-1e8243534e88 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a669fd0abc5c4554922f2693a217a301)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:46.945688 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b040b4-abe0-4576-859c-1e8243534e88 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:47.060898 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-92b040b4-abe0-4576-859c-1e8243534e88 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a669fd0abc5c4554922f2693a217a301'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:47.062940 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-92b040b4-abe0-4576-859c-1e8243534e88 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a669fd0abc5c4554922f2693a217a301'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:47.081139 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-92b040b4-abe0-4576-859c-1e8243534e88 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a669fd0abc5c4554922f2693a217a301'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:47.083757 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-92b040b4-abe0-4576-859c-1e8243534e88 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a669fd0abc5c4554922f2693a217a301'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:47.115695 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-92b040b4-abe0-4576-859c-1e8243534e88 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a669fd0abc5c4554922f2693a217a301'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:47.146166 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 824/1653] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:55:46 2022] DELETE /identity/v3/users/a669fd0abc5c4554922f2693a217a301 => generated 0 bytes in 468 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:55:49.294417 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d92ee7-48b3-4e9f-92b4-051300506721 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:49.297090 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-d2d92ee7-48b3-4e9f-92b4-051300506721 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:49.336088 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d92ee7-48b3-4e9f-92b4-051300506721 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:49.552651 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3cda9bb-c371-4131-bf65-1294982a3a24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:49.556359 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3cda9bb-c371-4131-bf65-1294982a3a24 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:49.556933 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3cda9bb-c371-4131-bf65-1294982a3a24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:49.557349 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3cda9bb-c371-4131-bf65-1294982a3a24 admin admin] PATH_INFO: `/v3/projects/966cbe725c7a45dda78e7ac72e56a7b1` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:49.558852 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3cda9bb-c371-4131-bf65-1294982a3a24 admin admin] RBAC: Authorizing `identity:delete_project(project_id=966cbe725c7a45dda78e7ac72e56a7b1)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:49.565044 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3cda9bb-c371-4131-bf65-1294982a3a24 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:49.567503 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:55:49.689518 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-a3cda9bb-c371-4131-bf65-1294982a3a24 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '966cbe725c7a45dda78e7ac72e56a7b1', 'request_id': 'req-a3cda9bb-c371-4131-bf65-1294982a3a24'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:49.712643 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-a3cda9bb-c371-4131-bf65-1294982a3a24 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '966cbe725c7a45dda78e7ac72e56a7b1', 'request_id': 'req-a3cda9bb-c371-4131-bf65-1294982a3a24'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:49.722243 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-a3cda9bb-c371-4131-bf65-1294982a3a24 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '966cbe725c7a45dda78e7ac72e56a7b1', 'request_id': 'req-a3cda9bb-c371-4131-bf65-1294982a3a24'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:49.751380 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-a3cda9bb-c371-4131-bf65-1294982a3a24 admin admin] The token cache is being invalidate because project 966cbe725c7a45dda78e7ac72e56a7b1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81386) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:55:49.751661 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-a3cda9bb-c371-4131-bf65-1294982a3a24 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:49.759034 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-a3cda9bb-c371-4131-bf65-1294982a3a24 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:49.761644 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 830/1654] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:55:49 2022] DELETE /identity/v3/projects/966cbe725c7a45dda78e7ac72e56a7b1 => generated 0 bytes in 469 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:55:49.767579 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c6a786-2f01-4432-943d-0f575d07c07b None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:49.771598 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c6a786-2f01-4432-943d-0f575d07c07b None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:49.771598 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c6a786-2f01-4432-943d-0f575d07c07b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:49.820120 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-28c6a786-2f01-4432-943d-0f575d07c07b None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:55:49.824160 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-28c6a786-2f01-4432-943d-0f575d07c07b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:49.847106 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3cda9bb-c371-4131-bf65-1294982a3a24 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:49.848943 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-a3cda9bb-c371-4131-bf65-1294982a3a24 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:49.953838 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:55:49.977989 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 825/1655] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:55:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 212 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:55:49.990249 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406907dc-c7af-4f57-bcf6-514ae93df911 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:49.990791 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406907dc-c7af-4f57-bcf6-514ae93df911 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:49.991159 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406907dc-c7af-4f57-bcf6-514ae93df911 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:50.026424 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-406907dc-c7af-4f57-bcf6-514ae93df911 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:55:50.027736 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-406907dc-c7af-4f57-bcf6-514ae93df911 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:50.032225 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3cda9bb-c371-4131-bf65-1294982a3a24 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:50.037059 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b574a29-5b8e-4d45-b009-b57eddab0279 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:50.037844 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b574a29-5b8e-4d45-b009-b57eddab0279 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:50.038273 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b574a29-5b8e-4d45-b009-b57eddab0279 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:50.038546 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b574a29-5b8e-4d45-b009-b57eddab0279 admin admin] PATH_INFO: `/v3/projects/cc00f38ca5bb4d34806d54bc8967f23c` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:50.041993 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b574a29-5b8e-4d45-b009-b57eddab0279 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cc00f38ca5bb4d34806d54bc8967f23c)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:50.042394 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 826/1656] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:55:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 54 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:55:50.047138 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-406907dc-c7af-4f57-bcf6-514ae93df911 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:50.049138 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b574a29-5b8e-4d45-b009-b57eddab0279 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:50.105791 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-406907dc-c7af-4f57-bcf6-514ae93df911 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:50.120003 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05239dac-31e2-4cc8-b81c-bca8879075cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:50.120644 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05239dac-31e2-4cc8-b81c-bca8879075cf admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:50.120922 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05239dac-31e2-4cc8-b81c-bca8879075cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:50.121193 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05239dac-31e2-4cc8-b81c-bca8879075cf admin admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:50.121882 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05239dac-31e2-4cc8-b81c-bca8879075cf admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:55:50.122185 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05239dac-31e2-4cc8-b81c-bca8879075cf admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:50.123876 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05239dac-31e2-4cc8-b81c-bca8879075cf admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:50.172399 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 827/1657] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 00:55:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 126 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:55:50.177037 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05239dac-31e2-4cc8-b81c-bca8879075cf admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:50.180152 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-5b574a29-5b8e-4d45-b009-b57eddab0279 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cc00f38ca5bb4d34806d54bc8967f23c', 'request_id': 'req-5b574a29-5b8e-4d45-b009-b57eddab0279'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:50.206003 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05239dac-31e2-4cc8-b81c-bca8879075cf admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:50.222869 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-5b574a29-5b8e-4d45-b009-b57eddab0279 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cc00f38ca5bb4d34806d54bc8967f23c', 'request_id': 'req-5b574a29-5b8e-4d45-b009-b57eddab0279'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:50.241875 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-5b574a29-5b8e-4d45-b009-b57eddab0279 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cc00f38ca5bb4d34806d54bc8967f23c', 'request_id': 'req-5b574a29-5b8e-4d45-b009-b57eddab0279'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:50.275674 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-5b574a29-5b8e-4d45-b009-b57eddab0279 admin admin] The token cache is being invalidate because project cc00f38ca5bb4d34806d54bc8967f23c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81386) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:55:50.275990 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-5b574a29-5b8e-4d45-b009-b57eddab0279 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:50.277091 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-5b574a29-5b8e-4d45-b009-b57eddab0279 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:50.288420 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 831/1658] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:55:49 2022] DELETE /identity/v3/projects/cc00f38ca5bb4d34806d54bc8967f23c => generated 0 bytes in 439 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 00:55:50.289943 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4810e1-b4a1-4826-84c2-e3c7cdab95f8 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:50.290348 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4810e1-b4a1-4826-84c2-e3c7cdab95f8 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:50.290829 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4810e1-b4a1-4826-84c2-e3c7cdab95f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:50.335441 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-6a4810e1-b4a1-4826-84c2-e3c7cdab95f8 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:55:50.338239 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-6a4810e1-b4a1-4826-84c2-e3c7cdab95f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:50.381756 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:55:50.417181 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29078eeb-25c0-4b55-a5f0-e963acd3326a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:50.419087 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29078eeb-25c0-4b55-a5f0-e963acd3326a admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:50.419368 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29078eeb-25c0-4b55-a5f0-e963acd3326a admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:50.419640 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29078eeb-25c0-4b55-a5f0-e963acd3326a admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:50.420823 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29078eeb-25c0-4b55-a5f0-e963acd3326a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:50.422230 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29078eeb-25c0-4b55-a5f0-e963acd3326a admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:50.479637 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 832/1659] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:55:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:55:50.482055 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a5eea1-89e2-4ab5-b315-ef7731c83f3d None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:50.482492 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a5eea1-89e2-4ab5-b315-ef7731c83f3d None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:50.482860 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a5eea1-89e2-4ab5-b315-ef7731c83f3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:50.504197 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 828/1660] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:55:50 2022] POST /identity/v3/projects => generated 363 bytes in 328 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:55:50.505223 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29078eeb-25c0-4b55-a5f0-e963acd3326a admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:50.508680 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-29078eeb-25c0-4b55-a5f0-e963acd3326a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:50.545743 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-f2a5eea1-89e2-4ab5-b315-ef7731c83f3d None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:55:50.548086 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-f2a5eea1-89e2-4ab5-b315-ef7731c83f3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:50.684208 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29078eeb-25c0-4b55-a5f0-e963acd3326a admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:50.690559 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-237e432f-e960-409b-a645-4773da90191b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:50.691178 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237e432f-e960-409b-a645-4773da90191b admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:50.691451 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237e432f-e960-409b-a645-4773da90191b admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:50.691716 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237e432f-e960-409b-a645-4773da90191b admin admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:50.692346 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-237e432f-e960-409b-a645-4773da90191b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:50.694309 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-237e432f-e960-409b-a645-4773da90191b admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:50.725873 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 829/1661] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:55:50 2022] POST /identity/v3/users => generated 282 bytes in 218 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:55:50.728265 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-237e432f-e960-409b-a645-4773da90191b admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:50.747736 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 833/1662] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:55:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 265 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:55:50.747736 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2a5eea1-89e2-4ab5-b315-ef7731c83f3d None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:50.754999 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-f2a5eea1-89e2-4ab5-b315-ef7731c83f3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:50.757813 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-237e432f-e960-409b-a645-4773da90191b admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:50.764608 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efdde3bc-b277-41aa-9111-309f3c38806f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:50.765629 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efdde3bc-b277-41aa-9111-309f3c38806f admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:50.766506 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efdde3bc-b277-41aa-9111-309f3c38806f admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:50.767079 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efdde3bc-b277-41aa-9111-309f3c38806f admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:50.768162 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efdde3bc-b277-41aa-9111-309f3c38806f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:50.771223 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efdde3bc-b277-41aa-9111-309f3c38806f admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:50.785393 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2a5eea1-89e2-4ab5-b315-ef7731c83f3d None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:50.790779 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 830/1663] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:55:50 2022] GET /identity/v3/roles => generated 1420 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 00:55:50.792365 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efdde3bc-b277-41aa-9111-309f3c38806f admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:50.796646 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f274b15-85a1-407f-a10d-1d3ee24d2ad2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:50.797488 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f274b15-85a1-407f-a10d-1d3ee24d2ad2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:50.797841 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f274b15-85a1-407f-a10d-1d3ee24d2ad2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:50.798259 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f274b15-85a1-407f-a10d-1d3ee24d2ad2 admin admin] PATH_INFO: `/v3/users/782f421af38b4c82a38ffc6ec1758b67` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:50.799759 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f274b15-85a1-407f-a10d-1d3ee24d2ad2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=782f421af38b4c82a38ffc6ec1758b67)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:50.803376 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: WARNING py.warnings [None req-8f274b15-85a1-407f-a10d-1d3ee24d2ad2 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 25 00:55:50.803376 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: warnings.warn(msg) Sep 25 00:55:50.803376 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: Sep 25 00:55:50.804976 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f274b15-85a1-407f-a10d-1d3ee24d2ad2 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:50.809399 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efdde3bc-b277-41aa-9111-309f3c38806f admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:50.822803 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b680d69f-60ab-4d5b-a154-c79756e71cf5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:50.823624 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b680d69f-60ab-4d5b-a154-c79756e71cf5 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:50.824105 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b680d69f-60ab-4d5b-a154-c79756e71cf5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:50.825028 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b680d69f-60ab-4d5b-a154-c79756e71cf5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:50.825910 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b680d69f-60ab-4d5b-a154-c79756e71cf5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:55:50.826518 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b680d69f-60ab-4d5b-a154-c79756e71cf5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:50.829318 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b680d69f-60ab-4d5b-a154-c79756e71cf5 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:50.853608 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 831/1664] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 00:55:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:55:50.855341 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b680d69f-60ab-4d5b-a154-c79756e71cf5 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:50.870461 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b680d69f-60ab-4d5b-a154-c79756e71cf5 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:50.877371 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbea8696-dfc9-4a3b-aa7f-464fab3727c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:50.878342 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbea8696-dfc9-4a3b-aa7f-464fab3727c0 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:50.880317 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbea8696-dfc9-4a3b-aa7f-464fab3727c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:50.880813 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbea8696-dfc9-4a3b-aa7f-464fab3727c0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:50.881721 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbea8696-dfc9-4a3b-aa7f-464fab3727c0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:50.885187 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbea8696-dfc9-4a3b-aa7f-464fab3727c0 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:50.900915 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 832/1665] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:55:50 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:55:50.906706 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbea8696-dfc9-4a3b-aa7f-464fab3727c0 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:50.924062 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbea8696-dfc9-4a3b-aa7f-464fab3727c0 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:50.949651 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6140390f-752c-4f6c-a856-f0023549b993 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:50.950546 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6140390f-752c-4f6c-a856-f0023549b993 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:50.950818 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6140390f-752c-4f6c-a856-f0023549b993 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:50.951087 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6140390f-752c-4f6c-a856-f0023549b993 admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:50.951961 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6140390f-752c-4f6c-a856-f0023549b993 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:50.955436 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6140390f-752c-4f6c-a856-f0023549b993 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:50.979735 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:55:51.003261 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 833/1666] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:55:50 2022] POST /identity/v3/projects => generated 383 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:55:51.004015 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-8f274b15-85a1-407f-a10d-1d3ee24d2ad2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '782f421af38b4c82a38ffc6ec1758b67'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:51.005153 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6140390f-752c-4f6c-a856-f0023549b993 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:51.023731 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-8f274b15-85a1-407f-a10d-1d3ee24d2ad2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '782f421af38b4c82a38ffc6ec1758b67'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:51.024846 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-8f274b15-85a1-407f-a10d-1d3ee24d2ad2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '782f421af38b4c82a38ffc6ec1758b67'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:51.059318 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-8f274b15-85a1-407f-a10d-1d3ee24d2ad2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '782f421af38b4c82a38ffc6ec1758b67'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:51.061822 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-8f274b15-85a1-407f-a10d-1d3ee24d2ad2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '782f421af38b4c82a38ffc6ec1758b67'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:51.138173 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 834/1667] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:55:50 2022] DELETE /identity/v3/users/782f421af38b4c82a38ffc6ec1758b67 => generated 0 bytes in 388 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:55:51.139318 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f274b15-85a1-407f-a10d-1d3ee24d2ad2 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:51.156025 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-8f274b15-85a1-407f-a10d-1d3ee24d2ad2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:51.167671 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6140390f-752c-4f6c-a856-f0023549b993 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:51.294594 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:55:51.350116 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed43b81f-03e4-487d-a725-8df73991a6e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:51.352993 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed43b81f-03e4-487d-a725-8df73991a6e2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:51.352993 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed43b81f-03e4-487d-a725-8df73991a6e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:51.352993 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed43b81f-03e4-487d-a725-8df73991a6e2 admin admin] PATH_INFO: `/v3/projects/122985cea56d4df18d25bf3e04482c9a/users/bc68d01ffa694cab800a3944117fd768/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:51.379954 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f274b15-85a1-407f-a10d-1d3ee24d2ad2 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:51.396490 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24201b1-dcae-4914-abfe-5b7644fe0755 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:51.397371 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24201b1-dcae-4914-abfe-5b7644fe0755 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:51.397658 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24201b1-dcae-4914-abfe-5b7644fe0755 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:51.398074 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24201b1-dcae-4914-abfe-5b7644fe0755 admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:51.399473 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c24201b1-dcae-4914-abfe-5b7644fe0755 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:51.401430 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed43b81f-03e4-487d-a725-8df73991a6e2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=122985cea56d4df18d25bf3e04482c9a, user_id=bc68d01ffa694cab800a3944117fd768, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:51.402954 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c24201b1-dcae-4914-abfe-5b7644fe0755 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:51.406196 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed43b81f-03e4-487d-a725-8df73991a6e2 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:51.450360 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 834/1668] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:55:51 2022] PUT /identity/v3/projects/122985cea56d4df18d25bf3e04482c9a/users/bc68d01ffa694cab800a3944117fd768/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 431 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:55:51.450360 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b7694b-6381-465d-8955-132e4640549a None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:51.450360 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b7694b-6381-465d-8955-132e4640549a None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:51.450360 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b7694b-6381-465d-8955-132e4640549a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:51.467594 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 835/1669] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:55:51 2022] POST /identity/v3/users => generated 292 bytes in 329 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:55:51.469384 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3875c21-ab20-4853-815b-8b646c078690 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:51.471399 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3875c21-ab20-4853-815b-8b646c078690 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:51.472364 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3875c21-ab20-4853-815b-8b646c078690 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:51.519820 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-29b7694b-6381-465d-8955-132e4640549a None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:55:51.530260 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-29b7694b-6381-465d-8955-132e4640549a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:51.566348 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-b3875c21-ab20-4853-815b-8b646c078690 None None] MFA Rules not processed for user `bc68d01ffa694cab800a3944117fd768`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:55:51.570550 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-b3875c21-ab20-4853-815b-8b646c078690 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:51.665883 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 835/1670] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:55:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 230 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:55:51.670725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac028049-ddca-44fd-b3e4-c2c0b9fa65d9 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:51.671010 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac028049-ddca-44fd-b3e4-c2c0b9fa65d9 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:51.671287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac028049-ddca-44fd-b3e4-c2c0b9fa65d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:51.691178 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 836/1671] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:55:51 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:55:51.692307 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3875c21-ab20-4853-815b-8b646c078690 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:51.717776 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3875c21-ab20-4853-815b-8b646c078690 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:51.732608 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30524370-770b-4ab9-9fed-c66b58617f79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:51.733581 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30524370-770b-4ab9-9fed-c66b58617f79 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:51.733937 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30524370-770b-4ab9-9fed-c66b58617f79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:51.734435 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30524370-770b-4ab9-9fed-c66b58617f79 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:51.734986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30524370-770b-4ab9-9fed-c66b58617f79 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:51.735477 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-ac028049-ddca-44fd-b3e4-c2c0b9fa65d9 None None] MFA Rules not processed for user `96386a16f0b54b7c94d32bd948caf118`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:55:51.736753 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-ac028049-ddca-44fd-b3e4-c2c0b9fa65d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:51.742611 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30524370-770b-4ab9-9fed-c66b58617f79 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:51.765811 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:55:51.770249 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 837/1672] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:55:51 2022] GET /identity/v3/roles => generated 1420 bytes in 77 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:55:51.771140 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30524370-770b-4ab9-9fed-c66b58617f79 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:51.777941 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-30524370-770b-4ab9-9fed-c66b58617f79 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:51.814310 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30524370-770b-4ab9-9fed-c66b58617f79 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:51.831742 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ebdeb8-cc6d-4466-b92a-573695dfbecd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:51.832946 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ebdeb8-cc6d-4466-b92a-573695dfbecd admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:51.833242 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ebdeb8-cc6d-4466-b92a-573695dfbecd admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:51.835472 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ebdeb8-cc6d-4466-b92a-573695dfbecd admin admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:51.835472 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7ebdeb8-cc6d-4466-b92a-573695dfbecd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:51.840876 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7ebdeb8-cc6d-4466-b92a-573695dfbecd admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:51.882803 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 836/1673] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:55:51 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 216 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:55:51.884237 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac028049-ddca-44fd-b3e4-c2c0b9fa65d9 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:51.907455 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac028049-ddca-44fd-b3e4-c2c0b9fa65d9 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:51.920291 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d6ecd8-bb5f-43c1-918b-b034c4d924f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:51.920499 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d6ecd8-bb5f-43c1-918b-b034c4d924f4 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:51.920745 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d6ecd8-bb5f-43c1-918b-b034c4d924f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:51.921015 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d6ecd8-bb5f-43c1-918b-b034c4d924f4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:51.921727 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d6ecd8-bb5f-43c1-918b-b034c4d924f4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:51.924552 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d6ecd8-bb5f-43c1-918b-b034c4d924f4 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:51.926039 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 838/1674] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:55:51 2022] POST /identity/v3/projects => generated 365 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:55:51.926561 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ebdeb8-cc6d-4466-b92a-573695dfbecd admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:51.928949 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-f7ebdeb8-cc6d-4466-b92a-573695dfbecd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:51.955829 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 837/1675] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:55:51 2022] GET /identity/v3/roles => generated 1420 bytes in 69 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:55:51.957860 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d6ecd8-bb5f-43c1-918b-b034c4d924f4 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:51.960580 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-75d6ecd8-bb5f-43c1-918b-b034c4d924f4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:52.092263 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d6ecd8-bb5f-43c1-918b-b034c4d924f4 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:52.121771 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ebdeb8-cc6d-4466-b92a-573695dfbecd admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:52.134579 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e7a108e-b188-453b-853a-30206e0aa99e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:52.134977 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7a108e-b188-453b-853a-30206e0aa99e service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:52.134977 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7a108e-b188-453b-853a-30206e0aa99e service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:52.135244 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7a108e-b188-453b-853a-30206e0aa99e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:52.231847 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:55:52.234499 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3013db13-fe1f-47fe-9e81-05ebfb37154d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f142988a4dd14a5f96eacea45c046f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:52.235135 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3013db13-fe1f-47fe-9e81-05ebfb37154d service nova] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:52.235415 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3013db13-fe1f-47fe-9e81-05ebfb37154d service nova] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:52.236289 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3013db13-fe1f-47fe-9e81-05ebfb37154d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:52.307232 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:55:52.310772 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e7a108e-b188-453b-853a-30206e0aa99e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:52.312790 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e7a108e-b188-453b-853a-30206e0aa99e service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:52.346613 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 839/1676] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:55:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 420 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:55:52.347754 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e7a108e-b188-453b-853a-30206e0aa99e service neutron] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:52.360007 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3013db13-fe1f-47fe-9e81-05ebfb37154d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:52.363061 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3013db13-fe1f-47fe-9e81-05ebfb37154d service nova] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:52.376313 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e7a108e-b188-453b-853a-30206e0aa99e service neutron] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:52.383747 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 838/1677] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:55:51 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 429 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:55:52.384746 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3013db13-fe1f-47fe-9e81-05ebfb37154d service nova] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:52.389838 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92df52dc-f79c-48f6-ac8c-0c91bcfab17a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:52.390359 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92df52dc-f79c-48f6-ac8c-0c91bcfab17a admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:52.390466 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92df52dc-f79c-48f6-ac8c-0c91bcfab17a admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:52.391611 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92df52dc-f79c-48f6-ac8c-0c91bcfab17a admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:52.391611 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92df52dc-f79c-48f6-ac8c-0c91bcfab17a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:52.396546 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92df52dc-f79c-48f6-ac8c-0c91bcfab17a admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:52.407365 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3013db13-fe1f-47fe-9e81-05ebfb37154d service nova] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:52.420428 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9d806cd-0239-4922-a5dc-0e63facd575e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:52.421952 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d806cd-0239-4922-a5dc-0e63facd575e admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:52.422358 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d806cd-0239-4922-a5dc-0e63facd575e admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:52.422664 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d806cd-0239-4922-a5dc-0e63facd575e admin admin] PATH_INFO: `/v3/projects/55b9b458194a4f4084ea2186c20b1efd/users/f105f660be9c42a895d6fd19262884cf/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:52.440679 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 840/1678] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:55:52 2022] POST /identity/v3/users => generated 289 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:55:52.444541 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92df52dc-f79c-48f6-ac8c-0c91bcfab17a admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:52.464387 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9d806cd-0239-4922-a5dc-0e63facd575e admin admin] RBAC: Authorizing `identity:create_grant(project_id=55b9b458194a4f4084ea2186c20b1efd, user_id=f105f660be9c42a895d6fd19262884cf, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:52.466143 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92df52dc-f79c-48f6-ac8c-0c91bcfab17a admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:52.469296 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9d806cd-0239-4922-a5dc-0e63facd575e admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:52.482321 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bda2627-e5b9-48e3-a542-92da8172bc59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:52.482957 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bda2627-e5b9-48e3-a542-92da8172bc59 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:52.483237 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bda2627-e5b9-48e3-a542-92da8172bc59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:52.483507 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bda2627-e5b9-48e3-a542-92da8172bc59 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:52.484167 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bda2627-e5b9-48e3-a542-92da8172bc59 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:52.485989 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bda2627-e5b9-48e3-a542-92da8172bc59 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:52.519570 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 841/1679] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:55:52 2022] GET /identity/v3/roles => generated 1420 bytes in 73 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 00:55:52.520536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bda2627-e5b9-48e3-a542-92da8172bc59 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:52.529594 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 839/1680] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:55:52 2022] PUT /identity/v3/projects/55b9b458194a4f4084ea2186c20b1efd/users/f105f660be9c42a895d6fd19262884cf/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:55:52.541368 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ed6dc9-672f-4f61-b5ce-c38d78a64802 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:52.542472 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ed6dc9-672f-4f61-b5ce-c38d78a64802 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:52.542920 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ed6dc9-672f-4f61-b5ce-c38d78a64802 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:52.544962 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bda2627-e5b9-48e3-a542-92da8172bc59 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:52.610684 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-01ed6dc9-672f-4f61-b5ce-c38d78a64802 None None] MFA Rules not processed for user `f105f660be9c42a895d6fd19262884cf`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:55:52.612226 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-01ed6dc9-672f-4f61-b5ce-c38d78a64802 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:52.692493 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c44e062c-f6dd-44c9-a6ac-e84dea3f1ca2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:52.693199 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44e062c-f6dd-44c9-a6ac-e84dea3f1ca2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:52.693512 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44e062c-f6dd-44c9-a6ac-e84dea3f1ca2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:52.693789 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44e062c-f6dd-44c9-a6ac-e84dea3f1ca2 admin admin] PATH_INFO: `/v3/projects/c549d8eee4e64e0196bb387d08150c6a/users/1e45257a614e4018b47f7292178979c9/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:52.732551 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c44e062c-f6dd-44c9-a6ac-e84dea3f1ca2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c549d8eee4e64e0196bb387d08150c6a, user_id=1e45257a614e4018b47f7292178979c9, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:52.735540 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c44e062c-f6dd-44c9-a6ac-e84dea3f1ca2 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:52.757673 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 842/1681] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:55:52 2022] PUT /identity/v3/projects/c549d8eee4e64e0196bb387d08150c6a/users/1e45257a614e4018b47f7292178979c9/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:55:52.759762 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c44e062c-f6dd-44c9-a6ac-e84dea3f1ca2 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:52.794234 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 840/1682] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:55:52 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 255 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 00:55:52.798938 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ed6dc9-672f-4f61-b5ce-c38d78a64802 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:52.813897 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ed6dc9-672f-4f61-b5ce-c38d78a64802 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:52.893184 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14748602-31e5-419c-bf5b-ae0a829caf82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:52.894825 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14748602-31e5-419c-bf5b-ae0a829caf82 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:52.895312 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14748602-31e5-419c-bf5b-ae0a829caf82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:52.895795 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14748602-31e5-419c-bf5b-ae0a829caf82 admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:52.896635 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14748602-31e5-419c-bf5b-ae0a829caf82 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:52.899006 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c44e062c-f6dd-44c9-a6ac-e84dea3f1ca2 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:52.900093 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14748602-31e5-419c-bf5b-ae0a829caf82 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:52.915417 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00fee94d-ff6a-46c1-a765-4c4e7945e4c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:52.916048 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00fee94d-ff6a-46c1-a765-4c4e7945e4c9 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:52.916324 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00fee94d-ff6a-46c1-a765-4c4e7945e4c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:52.916590 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00fee94d-ff6a-46c1-a765-4c4e7945e4c9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:52.917237 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00fee94d-ff6a-46c1-a765-4c4e7945e4c9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:52.918869 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00fee94d-ff6a-46c1-a765-4c4e7945e4c9 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:52.945359 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 841/1683] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:55:52 2022] POST /identity/v3/projects => generated 387 bytes in 147 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:55:52.948955 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14748602-31e5-419c-bf5b-ae0a829caf82 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:52.972549 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 843/1684] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:55:52 2022] GET /identity/v3/roles => generated 1420 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:55:52.973890 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00fee94d-ff6a-46c1-a765-4c4e7945e4c9 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:52.998338 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00fee94d-ff6a-46c1-a765-4c4e7945e4c9 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:52.999856 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:55:53.048684 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:55:53.062477 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14748602-31e5-419c-bf5b-ae0a829caf82 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:53.079065 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b869abf-a5b6-4e7a-b22a-9a23b2cc0143 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:53.080381 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b869abf-a5b6-4e7a-b22a-9a23b2cc0143 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:53.081756 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b869abf-a5b6-4e7a-b22a-9a23b2cc0143 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:53.082305 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b869abf-a5b6-4e7a-b22a-9a23b2cc0143 admin admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:53.083701 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b869abf-a5b6-4e7a-b22a-9a23b2cc0143 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:53.090829 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b869abf-a5b6-4e7a-b22a-9a23b2cc0143 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:53.144031 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 842/1685] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:55:52 2022] POST /identity/v3/users => generated 294 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:55:53.147881 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b869abf-a5b6-4e7a-b22a-9a23b2cc0143 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:53.148609 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf18c859-5604-4a6f-a999-697bff90b147 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:53.149914 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf18c859-5604-4a6f-a999-697bff90b147 admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:53.150214 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf18c859-5604-4a6f-a999-697bff90b147 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:53.150473 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf18c859-5604-4a6f-a999-697bff90b147 admin admin] PATH_INFO: `/v3/projects/c549d8eee4e64e0196bb387d08150c6a/users/1e45257a614e4018b47f7292178979c9/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:53.157322 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf18c859-5604-4a6f-a999-697bff90b147 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c549d8eee4e64e0196bb387d08150c6a, user_id=1e45257a614e4018b47f7292178979c9, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:53.161960 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf18c859-5604-4a6f-a999-697bff90b147 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:53.169606 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b869abf-a5b6-4e7a-b22a-9a23b2cc0143 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:53.182020 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d0f735b-fad2-49e2-92cf-24227a078e99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:53.182557 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0f735b-fad2-49e2-92cf-24227a078e99 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:53.182832 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0f735b-fad2-49e2-92cf-24227a078e99 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:53.183505 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0f735b-fad2-49e2-92cf-24227a078e99 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:53.183806 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d0f735b-fad2-49e2-92cf-24227a078e99 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:53.197357 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 844/1686] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:55:52 2022] PUT /identity/v3/projects/c549d8eee4e64e0196bb387d08150c6a/users/1e45257a614e4018b47f7292178979c9/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 00:55:53.197666 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d0f735b-fad2-49e2-92cf-24227a078e99 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:53.201916 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4604fb5e-8bf7-4659-a631-95f829634896 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:53.202201 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4604fb5e-8bf7-4659-a631-95f829634896 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:53.202472 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4604fb5e-8bf7-4659-a631-95f829634896 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:53.215506 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 843/1687] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:55:53 2022] GET /identity/v3/roles => generated 1420 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:55:53.218317 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d0f735b-fad2-49e2-92cf-24227a078e99 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:53.257025 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d0f735b-fad2-49e2-92cf-24227a078e99 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:53.310183 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-4604fb5e-8bf7-4659-a631-95f829634896 None None] MFA Rules not processed for user `1e45257a614e4018b47f7292178979c9`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:55:53.315138 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-4604fb5e-8bf7-4659-a631-95f829634896 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:53.509609 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fda4588-5891-4854-8baf-267768de38ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:53.512761 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fda4588-5891-4854-8baf-267768de38ed admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:53.512761 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fda4588-5891-4854-8baf-267768de38ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:53.512761 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fda4588-5891-4854-8baf-267768de38ed admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:53.514824 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fda4588-5891-4854-8baf-267768de38ed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:53.517844 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fda4588-5891-4854-8baf-267768de38ed admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:53.543614 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 844/1688] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:55:53 2022] GET /identity/v3/roles => generated 1420 bytes in 325 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:55:53.554813 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fda4588-5891-4854-8baf-267768de38ed admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:53.571793 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 845/1689] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:55:53 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 368 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:55:53.580017 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4604fb5e-8bf7-4659-a631-95f829634896 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:53.581635 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-4604fb5e-8bf7-4659-a631-95f829634896 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:53.584392 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fda4588-5891-4854-8baf-267768de38ed admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:53.599058 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c58fe8b-3a32-41bc-b388-f9f56bed61f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:53.600588 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c58fe8b-3a32-41bc-b388-f9f56bed61f4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:53.601048 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c58fe8b-3a32-41bc-b388-f9f56bed61f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:53.602203 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c58fe8b-3a32-41bc-b388-f9f56bed61f4 admin admin] PATH_INFO: `/v3/projects/8928bfafbc184624805c19c6ecd97656/users/5aad523f24fa4be5a28b25a4a1bbe56f/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:53.631365 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c58fe8b-3a32-41bc-b388-f9f56bed61f4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8928bfafbc184624805c19c6ecd97656, user_id=5aad523f24fa4be5a28b25a4a1bbe56f, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:53.636288 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c58fe8b-3a32-41bc-b388-f9f56bed61f4 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:53.664785 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 845/1690] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:55:53 2022] PUT /identity/v3/projects/8928bfafbc184624805c19c6ecd97656/users/5aad523f24fa4be5a28b25a4a1bbe56f/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 00:55:53.671043 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4604fb5e-8bf7-4659-a631-95f829634896 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:53.674603 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6910a7b1-9b72-467c-8f46-6ebf255c1ab6 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:53.675028 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6910a7b1-9b72-467c-8f46-6ebf255c1ab6 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:53.675370 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6910a7b1-9b72-467c-8f46-6ebf255c1ab6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:53.747964 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-6910a7b1-9b72-467c-8f46-6ebf255c1ab6 None None] MFA Rules not processed for user `5aad523f24fa4be5a28b25a4a1bbe56f`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:55:53.749217 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-6910a7b1-9b72-467c-8f46-6ebf255c1ab6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:53.798928 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf18af1-880b-47bf-b3c4-9f13b94bd536 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:53.799570 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf18af1-880b-47bf-b3c4-9f13b94bd536 service cinder] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:53.799844 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf18af1-880b-47bf-b3c4-9f13b94bd536 service cinder] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:53.801136 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf18af1-880b-47bf-b3c4-9f13b94bd536 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:53.817125 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:55:53.903365 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf18af1-880b-47bf-b3c4-9f13b94bd536 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:53.904853 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf18af1-880b-47bf-b3c4-9f13b94bd536 service cinder] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:53.912500 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 846/1691] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:55:53 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 239 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 00:55:53.913502 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6910a7b1-9b72-467c-8f46-6ebf255c1ab6 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:53.919518 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 846/1692] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:55:53 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 340 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:55:53.929730 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:55:54.054378 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6910a7b1-9b72-467c-8f46-6ebf255c1ab6 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:54.056082 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf18af1-880b-47bf-b3c4-9f13b94bd536 service cinder] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:54.062065 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ea383d-4775-4839-868e-76e5b9039795 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:54.062704 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ea383d-4775-4839-868e-76e5b9039795 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:54.062983 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ea383d-4775-4839-868e-76e5b9039795 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:54.064318 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ea383d-4775-4839-868e-76e5b9039795 admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:54.064620 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7ea383d-4775-4839-868e-76e5b9039795 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:54.066599 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7ea383d-4775-4839-868e-76e5b9039795 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:54.073072 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf18af1-880b-47bf-b3c4-9f13b94bd536 service cinder] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:54.084819 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e90318e-b8c7-4558-9325-4d43aea2e7f0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:54.085741 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e90318e-b8c7-4558-9325-4d43aea2e7f0 service cinder] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:54.087247 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e90318e-b8c7-4558-9325-4d43aea2e7f0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:54.089494 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e90318e-b8c7-4558-9325-4d43aea2e7f0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:54.151083 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 847/1693] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:55:53 2022] POST /identity/v3/projects => generated 387 bytes in 238 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:55:54.152310 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ea383d-4775-4839-868e-76e5b9039795 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:54.231900 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e90318e-b8c7-4558-9325-4d43aea2e7f0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:54.233503 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e90318e-b8c7-4558-9325-4d43aea2e7f0 service cinder] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:54.263579 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 847/1694] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:55:54 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 209 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:55:54.308909 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ea383d-4775-4839-868e-76e5b9039795 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:54.323674 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-874b900a-5ee9-4844-b746-510c5946afad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:54.324111 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874b900a-5ee9-4844-b746-510c5946afad admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:54.324394 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874b900a-5ee9-4844-b746-510c5946afad admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:54.324773 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874b900a-5ee9-4844-b746-510c5946afad admin admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:54.325316 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-874b900a-5ee9-4844-b746-510c5946afad admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:54.327275 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-874b900a-5ee9-4844-b746-510c5946afad admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:54.372453 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 848/1695] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:55:54 2022] POST /identity/v3/users => generated 300 bytes in 220 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:55:54.377068 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e90318e-b8c7-4558-9325-4d43aea2e7f0 service cinder] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:54.397009 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e90318e-b8c7-4558-9325-4d43aea2e7f0 service cinder] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:54.423225 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27fb9b7e-3ba3-46cf-b399-f1d6c131fd54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:54.423860 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fb9b7e-3ba3-46cf-b399-f1d6c131fd54 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:54.424136 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fb9b7e-3ba3-46cf-b399-f1d6c131fd54 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:54.424407 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fb9b7e-3ba3-46cf-b399-f1d6c131fd54 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:54.425060 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27fb9b7e-3ba3-46cf-b399-f1d6c131fd54 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:54.426725 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27fb9b7e-3ba3-46cf-b399-f1d6c131fd54 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:54.450803 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 848/1696] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:55:54 2022] GET /identity/v3/roles => generated 1420 bytes in 75 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:55:54.457178 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-874b900a-5ee9-4844-b746-510c5946afad admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:54.490113 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-874b900a-5ee9-4844-b746-510c5946afad admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:54.511315 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e82b03-95d1-4eaf-a70f-c5eb0e614358 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:54.512199 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e82b03-95d1-4eaf-a70f-c5eb0e614358 admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:54.512617 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e82b03-95d1-4eaf-a70f-c5eb0e614358 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:54.513028 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e82b03-95d1-4eaf-a70f-c5eb0e614358 admin admin] PATH_INFO: `/v3/projects/a0aa105c58b1474cbe337ec83f25fdfc/users/8078163880b6454cbe8b6bce54b52ad0/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:54.552778 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e82b03-95d1-4eaf-a70f-c5eb0e614358 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a0aa105c58b1474cbe337ec83f25fdfc, user_id=8078163880b6454cbe8b6bce54b52ad0, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:54.562669 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e82b03-95d1-4eaf-a70f-c5eb0e614358 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:54.617184 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27fb9b7e-3ba3-46cf-b399-f1d6c131fd54 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:54.619406 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 849/1697] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:55:54 2022] PUT /identity/v3/projects/a0aa105c58b1474cbe337ec83f25fdfc/users/8078163880b6454cbe8b6bce54b52ad0/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:55:54.623172 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-27fb9b7e-3ba3-46cf-b399-f1d6c131fd54 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:54.628344 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e82b03-95d1-4eaf-a70f-c5eb0e614358 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:54.647000 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27fb9b7e-3ba3-46cf-b399-f1d6c131fd54 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:54.819775 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e82b03-95d1-4eaf-a70f-c5eb0e614358 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:54.824386 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c3947c-b109-453f-827a-900fbae2ebbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:54.825036 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c3947c-b109-453f-827a-900fbae2ebbb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:54.825331 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c3947c-b109-453f-827a-900fbae2ebbb admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:54.825601 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c3947c-b109-453f-827a-900fbae2ebbb admin admin] PATH_INFO: `/v3/users/4fad770ae27641b3beb2be9f9c4673b6` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:54.827241 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54c3947c-b109-453f-827a-900fbae2ebbb admin admin] RBAC: Authorizing `identity:delete_user(user_id=4fad770ae27641b3beb2be9f9c4673b6)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:54.829976 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54c3947c-b109-453f-827a-900fbae2ebbb admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:54.837751 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29036ab-f235-4d0f-b67d-f27bd8599b2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:54.838377 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29036ab-f235-4d0f-b67d-f27bd8599b2f admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:54.838661 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29036ab-f235-4d0f-b67d-f27bd8599b2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:54.840073 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29036ab-f235-4d0f-b67d-f27bd8599b2f admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:54.840073 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d29036ab-f235-4d0f-b67d-f27bd8599b2f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:54.841170 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d29036ab-f235-4d0f-b67d-f27bd8599b2f admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:54.846746 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:55:54.872505 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 850/1698] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:55:54 2022] GET /identity/v3/roles => generated 1420 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:55:54.873555 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29036ab-f235-4d0f-b67d-f27bd8599b2f admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:54.880973 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-d29036ab-f235-4d0f-b67d-f27bd8599b2f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:54.912253 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:55:54.915732 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29036ab-f235-4d0f-b67d-f27bd8599b2f admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:54.927694 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42b926d6-01c6-4eb5-a59d-fe62404adf73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:54.929219 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b926d6-01c6-4eb5-a59d-fe62404adf73 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:54.932189 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b926d6-01c6-4eb5-a59d-fe62404adf73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:54.932189 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b926d6-01c6-4eb5-a59d-fe62404adf73 admin admin] PATH_INFO: `/v3/projects/1bc1cab13abc43779d57421107671bdf` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:54.934448 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42b926d6-01c6-4eb5-a59d-fe62404adf73 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1bc1cab13abc43779d57421107671bdf)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:54.935010 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-54c3947c-b109-453f-827a-900fbae2ebbb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4fad770ae27641b3beb2be9f9c4673b6'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:54.936158 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: WARNING py.warnings [None req-42b926d6-01c6-4eb5-a59d-fe62404adf73 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 25 00:55:54.936158 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: warnings.warn(msg) Sep 25 00:55:54.936158 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: Sep 25 00:55:54.940520 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42b926d6-01c6-4eb5-a59d-fe62404adf73 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:54.950971 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-54c3947c-b109-453f-827a-900fbae2ebbb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4fad770ae27641b3beb2be9f9c4673b6'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:54.951829 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-54c3947c-b109-453f-827a-900fbae2ebbb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4fad770ae27641b3beb2be9f9c4673b6'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:54.979810 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-54c3947c-b109-453f-827a-900fbae2ebbb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4fad770ae27641b3beb2be9f9c4673b6'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:54.983289 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-54c3947c-b109-453f-827a-900fbae2ebbb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4fad770ae27641b3beb2be9f9c4673b6'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:55.036127 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 849/1699] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:55:54 2022] DELETE /identity/v3/users/4fad770ae27641b3beb2be9f9c4673b6 => generated 0 bytes in 419 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:55:55.039561 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c3947c-b109-453f-827a-900fbae2ebbb admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:55.042624 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-54c3947c-b109-453f-827a-900fbae2ebbb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:55.120608 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-42b926d6-01c6-4eb5-a59d-fe62404adf73 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1bc1cab13abc43779d57421107671bdf', 'request_id': 'req-42b926d6-01c6-4eb5-a59d-fe62404adf73'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:55.138060 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-42b926d6-01c6-4eb5-a59d-fe62404adf73 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1bc1cab13abc43779d57421107671bdf', 'request_id': 'req-42b926d6-01c6-4eb5-a59d-fe62404adf73'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:55.151913 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-42b926d6-01c6-4eb5-a59d-fe62404adf73 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1bc1cab13abc43779d57421107671bdf', 'request_id': 'req-42b926d6-01c6-4eb5-a59d-fe62404adf73'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:55.212357 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-42b926d6-01c6-4eb5-a59d-fe62404adf73 admin admin] The token cache is being invalidate because project 1bc1cab13abc43779d57421107671bdf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:55:55.212668 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-42b926d6-01c6-4eb5-a59d-fe62404adf73 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:55.214509 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-42b926d6-01c6-4eb5-a59d-fe62404adf73 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:55:55.216790 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 851/1700] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:55:54 2022] DELETE /identity/v3/projects/1bc1cab13abc43779d57421107671bdf => generated 0 bytes in 344 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:55:55.220187 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1211c0e-56b5-4cde-b3cd-ef4fec483bec None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:55.220187 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1211c0e-56b5-4cde-b3cd-ef4fec483bec None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:55.220187 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1211c0e-56b5-4cde-b3cd-ef4fec483bec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:55.240170 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c3947c-b109-453f-827a-900fbae2ebbb admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:55.268171 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-d1211c0e-56b5-4cde-b3cd-ef4fec483bec None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:55:55.270904 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-d1211c0e-56b5-4cde-b3cd-ef4fec483bec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:55.396953 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22d76380-cf93-44b9-9303-01afcfb890de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:55.398044 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d76380-cf93-44b9-9303-01afcfb890de admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:55.399591 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d76380-cf93-44b9-9303-01afcfb890de admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:55.400114 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d76380-cf93-44b9-9303-01afcfb890de admin admin] PATH_INFO: `/v3/projects/a0aa105c58b1474cbe337ec83f25fdfc/users/8078163880b6454cbe8b6bce54b52ad0/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:55.402584 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22d76380-cf93-44b9-9303-01afcfb890de admin admin] RBAC: Authorizing `identity:create_grant(project_id=a0aa105c58b1474cbe337ec83f25fdfc, user_id=8078163880b6454cbe8b6bce54b52ad0, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:55.407717 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22d76380-cf93-44b9-9303-01afcfb890de admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:55.418939 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 852/1701] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:55:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:55:55.420114 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c5e3f2-ceb5-4184-bd93-53b95206226b None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:55.420401 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c5e3f2-ceb5-4184-bd93-53b95206226b None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:55.423460 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c5e3f2-ceb5-4184-bd93-53b95206226b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:55.439854 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 850/1702] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:55:55 2022] PUT /identity/v3/projects/a0aa105c58b1474cbe337ec83f25fdfc/users/8078163880b6454cbe8b6bce54b52ad0/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 403 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:55:55.453928 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0546f0-4eb7-4563-86f8-2ac9e1a0b8d9 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:55.454122 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0546f0-4eb7-4563-86f8-2ac9e1a0b8d9 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:55.454331 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0546f0-4eb7-4563-86f8-2ac9e1a0b8d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:55.507417 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-c4c5e3f2-ceb5-4184-bd93-53b95206226b None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:55:55.515823 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-c4c5e3f2-ceb5-4184-bd93-53b95206226b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:55.531636 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-3a0546f0-4eb7-4563-86f8-2ac9e1a0b8d9 None None] MFA Rules not processed for user `8078163880b6454cbe8b6bce54b52ad0`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:55:55.532884 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-3a0546f0-4eb7-4563-86f8-2ac9e1a0b8d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:55.627366 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=93,keystone:DELETE=5,keystone:INSERT=2 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:55:55.687322 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 851/1703] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:55:55 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 234 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 00:55:55.688225 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a0546f0-4eb7-4563-86f8-2ac9e1a0b8d9 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:55.697398 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-3a0546f0-4eb7-4563-86f8-2ac9e1a0b8d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:55.719456 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 853/1704] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:55:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 300 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:55:55.721469 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d003608-0766-4c1b-8f7b-a6b5de80b350 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:55.722039 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d003608-0766-4c1b-8f7b-a6b5de80b350 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:55.722473 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d003608-0766-4c1b-8f7b-a6b5de80b350 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:55.814162 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-5d003608-0766-4c1b-8f7b-a6b5de80b350 None None] MFA Rules not processed for user `8eabe2a74a604c47bc2e5c88b960dcb6`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:55:55.822112 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-5d003608-0766-4c1b-8f7b-a6b5de80b350 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:55.853511 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:55:55.910975 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a0546f0-4eb7-4563-86f8-2ac9e1a0b8d9 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:55.949837 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d26a79c-67fc-4aea-8012-28b477fe68ee service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:55.950481 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d26a79c-67fc-4aea-8012-28b477fe68ee service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:55.950762 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d26a79c-67fc-4aea-8012-28b477fe68ee service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:55.951028 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d26a79c-67fc-4aea-8012-28b477fe68ee service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:55.982143 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d26a79c-67fc-4aea-8012-28b477fe68ee service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:55.983640 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d26a79c-67fc-4aea-8012-28b477fe68ee service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:56.001734 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 852/1705] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:55:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:55:56.002953 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d26a79c-67fc-4aea-8012-28b477fe68ee service neutron] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:56.003980 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-4d26a79c-67fc-4aea-8012-28b477fe68ee service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:56.041851 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 854/1706] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:55:55 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 322 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:55:56.043221 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4773f4b-1920-4055-a952-4fc30a2526fc None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:56.043511 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4773f4b-1920-4055-a952-4fc30a2526fc None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:56.043779 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4773f4b-1920-4055-a952-4fc30a2526fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:56.100375 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-d4773f4b-1920-4055-a952-4fc30a2526fc None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:55:56.108389 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-d4773f4b-1920-4055-a952-4fc30a2526fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:56.127181 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d26a79c-67fc-4aea-8012-28b477fe68ee service neutron] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:56.148598 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 855/1707] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:55:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:55:56.152581 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30b7aa3-0b64-464a-87df-e0569630320e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:56.152764 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e47949ce-a061-4272-9574-b5c2c928de89 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:56.153481 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30b7aa3-0b64-464a-87df-e0569630320e service cinder] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:56.154005 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30b7aa3-0b64-464a-87df-e0569630320e service cinder] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:56.154373 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30b7aa3-0b64-464a-87df-e0569630320e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:56.155142 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e47949ce-a061-4272-9574-b5c2c928de89 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:56.155425 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e47949ce-a061-4272-9574-b5c2c928de89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:56.174896 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d30b7aa3-0b64-464a-87df-e0569630320e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:56.176687 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d30b7aa3-0b64-464a-87df-e0569630320e service cinder] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:56.213215 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 853/1708] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:55:56 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 210 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:55:56.214631 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30b7aa3-0b64-464a-87df-e0569630320e service cinder] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:56.222204 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-e47949ce-a061-4272-9574-b5c2c928de89 None None] MFA Rules not processed for user `8eabe2a74a604c47bc2e5c88b960dcb6`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:55:56.223465 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-e47949ce-a061-4272-9574-b5c2c928de89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:56.245186 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 856/1709] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:55:56 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:55:56.257246 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e47949ce-a061-4272-9574-b5c2c928de89 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:56.259316 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-e47949ce-a061-4272-9574-b5c2c928de89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:56.282106 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30b7aa3-0b64-464a-87df-e0569630320e service cinder] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:56.294592 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62dc345a-347b-4a55-91d0-cd77c0e47c6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:56.295317 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62dc345a-347b-4a55-91d0-cd77c0e47c6e admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:56.297239 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62dc345a-347b-4a55-91d0-cd77c0e47c6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:56.297239 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62dc345a-347b-4a55-91d0-cd77c0e47c6e admin admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:56.298937 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62dc345a-347b-4a55-91d0-cd77c0e47c6e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:55:56.300391 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62dc345a-347b-4a55-91d0-cd77c0e47c6e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:56.303632 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62dc345a-347b-4a55-91d0-cd77c0e47c6e admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:56.334882 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 854/1710] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 00:55:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 121 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:55:56.346295 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62dc345a-347b-4a55-91d0-cd77c0e47c6e admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:56.374622 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62dc345a-347b-4a55-91d0-cd77c0e47c6e admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:56.388636 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415cf9e9-39d0-48ae-b740-69f66e154a51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:56.389648 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415cf9e9-39d0-48ae-b740-69f66e154a51 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:56.390166 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415cf9e9-39d0-48ae-b740-69f66e154a51 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:56.390630 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415cf9e9-39d0-48ae-b740-69f66e154a51 admin admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:56.391473 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-415cf9e9-39d0-48ae-b740-69f66e154a51 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:56.394714 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-415cf9e9-39d0-48ae-b740-69f66e154a51 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:56.448397 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e47949ce-a061-4272-9574-b5c2c928de89 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:56.475576 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 855/1711] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:55:56 2022] POST /identity/v3/projects => generated 371 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:55:56.475862 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175561aa-1c69-4e82-9c94-566dcb51bcdc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f142988a4dd14a5f96eacea45c046f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:56.476652 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415cf9e9-39d0-48ae-b740-69f66e154a51 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:56.476760 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175561aa-1c69-4e82-9c94-566dcb51bcdc service nova] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:56.476886 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175561aa-1c69-4e82-9c94-566dcb51bcdc service nova] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:56.493623 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175561aa-1c69-4e82-9c94-566dcb51bcdc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:56.537071 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-175561aa-1c69-4e82-9c94-566dcb51bcdc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:56.545027 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-175561aa-1c69-4e82-9c94-566dcb51bcdc service nova] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:56.583527 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 857/1712] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:55:56 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 327 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 00:55:56.670423 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:55:56.710714 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415cf9e9-39d0-48ae-b740-69f66e154a51 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:56.734309 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9721b32-8aa6-4741-af04-c46b18e2b8bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:56.734946 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9721b32-8aa6-4741-af04-c46b18e2b8bd admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:56.735226 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9721b32-8aa6-4741-af04-c46b18e2b8bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:56.735488 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9721b32-8aa6-4741-af04-c46b18e2b8bd admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:56.736132 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9721b32-8aa6-4741-af04-c46b18e2b8bd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:56.745843 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9721b32-8aa6-4741-af04-c46b18e2b8bd admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:56.802587 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 856/1713] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:55:56 2022] POST /identity/v3/users => generated 286 bytes in 326 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:55:56.807175 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175561aa-1c69-4e82-9c94-566dcb51bcdc service nova] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:56.843928 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175561aa-1c69-4e82-9c94-566dcb51bcdc service nova] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:56.870964 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb1d36e-59d8-47ad-8246-dffba2c04bed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:56.874835 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb1d36e-59d8-47ad-8246-dffba2c04bed admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:56.893074 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb1d36e-59d8-47ad-8246-dffba2c04bed admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:56.900015 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb1d36e-59d8-47ad-8246-dffba2c04bed admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:56.901114 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb1d36e-59d8-47ad-8246-dffba2c04bed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:56.909915 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb1d36e-59d8-47ad-8246-dffba2c04bed admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:56.973130 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 858/1714] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:55:56 2022] GET /identity/v3/roles => generated 1420 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 00:55:56.974639 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9721b32-8aa6-4741-af04-c46b18e2b8bd admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:56.993582 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9721b32-8aa6-4741-af04-c46b18e2b8bd admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:57.014956 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3942978a-2db9-4e34-b8cb-4f0a359ab566 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:57.015660 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3942978a-2db9-4e34-b8cb-4f0a359ab566 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:57.017558 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3942978a-2db9-4e34-b8cb-4f0a359ab566 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:57.017902 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3942978a-2db9-4e34-b8cb-4f0a359ab566 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:57.019096 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3942978a-2db9-4e34-b8cb-4f0a359ab566 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:57.024922 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3942978a-2db9-4e34-b8cb-4f0a359ab566 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:57.063849 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb1d36e-59d8-47ad-8246-dffba2c04bed admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:57.070748 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 857/1715] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:55:56 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:55:57.078186 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb1d36e-59d8-47ad-8246-dffba2c04bed admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:57.089210 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc2c3213-1d1b-43d8-ab1c-e886cfaa7498 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:57.089944 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc2c3213-1d1b-43d8-ab1c-e886cfaa7498 admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:57.090218 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc2c3213-1d1b-43d8-ab1c-e886cfaa7498 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:57.090490 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc2c3213-1d1b-43d8-ab1c-e886cfaa7498 admin admin] PATH_INFO: `/v3/projects/03e39e5230b1454197fb977b74b18596/users/a48e90a2614d4e098260868a34bb2af2/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:57.128198 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc2c3213-1d1b-43d8-ab1c-e886cfaa7498 admin admin] RBAC: Authorizing `identity:create_grant(project_id=03e39e5230b1454197fb977b74b18596, user_id=a48e90a2614d4e098260868a34bb2af2, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:57.131261 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc2c3213-1d1b-43d8-ab1c-e886cfaa7498 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:57.178837 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 859/1716] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:55:57 2022] PUT /identity/v3/projects/03e39e5230b1454197fb977b74b18596/users/a48e90a2614d4e098260868a34bb2af2/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 00:55:57.192099 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d51660-9792-4db2-a476-bd4b361efa4f None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:57.192304 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d51660-9792-4db2-a476-bd4b361efa4f None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:57.192602 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d51660-9792-4db2-a476-bd4b361efa4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:57.260495 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-f5d51660-9792-4db2-a476-bd4b361efa4f None None] MFA Rules not processed for user `a48e90a2614d4e098260868a34bb2af2`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:55:57.262944 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-f5d51660-9792-4db2-a476-bd4b361efa4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:57.483888 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 858/1717] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:55:57 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 294 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:55:57.500022 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc2c3213-1d1b-43d8-ab1c-e886cfaa7498 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:57.503576 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-fc2c3213-1d1b-43d8-ab1c-e886cfaa7498 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:57.635070 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:55:57.712930 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc2c3213-1d1b-43d8-ab1c-e886cfaa7498 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:57.727028 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5d51660-9792-4db2-a476-bd4b361efa4f None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:55:57.755275 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8304217-673c-449a-a68e-9c594a562cc3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f142988a4dd14a5f96eacea45c046f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:57.755909 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8304217-673c-449a-a68e-9c594a562cc3 service nova] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:57.756236 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8304217-673c-449a-a68e-9c594a562cc3 service nova] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:57.756666 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8304217-673c-449a-a68e-9c594a562cc3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:57.788122 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8304217-673c-449a-a68e-9c594a562cc3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:57.791715 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8304217-673c-449a-a68e-9c594a562cc3 service nova] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:57.835422 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 860/1718] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:55:57 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 337 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:55:57.865167 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5d51660-9792-4db2-a476-bd4b361efa4f None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:55:57.872466 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5966c6e4-2931-426e-95eb-b09d75d4d98b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:55:57.873162 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5966c6e4-2931-426e-95eb-b09d75d4d98b service cinder] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:57.873603 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5966c6e4-2931-426e-95eb-b09d75d4d98b service cinder] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:57.873942 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5966c6e4-2931-426e-95eb-b09d75d4d98b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:57.876281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-5966c6e4-2931-426e-95eb-b09d75d4d98b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:58.142086 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5966c6e4-2931-426e-95eb-b09d75d4d98b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:55:58.154780 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5966c6e4-2931-426e-95eb-b09d75d4d98b service cinder] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:55:58.187373 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 859/1719] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:55:57 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 461 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:55:58.915859 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368947eb-5947-46fb-8a17-41d9e23dd936 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:58.916404 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368947eb-5947-46fb-8a17-41d9e23dd936 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:58.916404 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368947eb-5947-46fb-8a17-41d9e23dd936 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:58.970131 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-368947eb-5947-46fb-8a17-41d9e23dd936 None None] MFA Rules not processed for user `96386a16f0b54b7c94d32bd948caf118`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:55:58.971792 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-368947eb-5947-46fb-8a17-41d9e23dd936 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:59.210415 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 861/1720] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:55:58 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 296 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:55:59.563669 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32005e4e-7d8e-431d-af8b-0e090bbfbbd2 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:59.564582 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32005e4e-7d8e-431d-af8b-0e090bbfbbd2 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:59.565201 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32005e4e-7d8e-431d-af8b-0e090bbfbbd2 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:59.567842 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 860/1721] 10.222.0.52 () {54 vars in 912 bytes} [Sun Sep 25 00:55:59 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:55:59.575307 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092a7176-be68-43fa-bbd5-dbdc4e8e8574 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:55:59.575607 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092a7176-be68-43fa-bbd5-dbdc4e8e8574 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:55:59.575891 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092a7176-be68-43fa-bbd5-dbdc4e8e8574 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:55:59.725594 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-092a7176-be68-43fa-bbd5-dbdc4e8e8574 None None] MFA Rules not processed for user `20c274e96dec478b98ef51f80ea893b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:55:59.727364 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-092a7176-be68-43fa-bbd5-dbdc4e8e8574 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:55:59.908559 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 862/1722] 10.222.0.52 () {58 vars in 1009 bytes} [Sun Sep 25 00:55:59 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 335 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:00.113574 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32005e4e-7d8e-431d-af8b-0e090bbfbbd2 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:00.145668 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32005e4e-7d8e-431d-af8b-0e090bbfbbd2 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:00.153811 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c83035-500d-4033-ac03-a303a4a6cb3b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:00.157307 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c83035-500d-4033-ac03-a303a4a6cb3b service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:00.157781 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c83035-500d-4033-ac03-a303a4a6cb3b service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:00.158321 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c83035-500d-4033-ac03-a303a4a6cb3b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:00.171961 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6c83035-500d-4033-ac03-a303a4a6cb3b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:00.173761 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6c83035-500d-4033-ac03-a303a4a6cb3b service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:00.180837 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:00.192039 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 861/1723] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:56:00 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 79 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:56:00.270815 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae79333-8fbb-46d2-9381-15a6f0521299 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:00.272411 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae79333-8fbb-46d2-9381-15a6f0521299 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:00.273314 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae79333-8fbb-46d2-9381-15a6f0521299 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:00.277572 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 863/1724] 10.222.0.52 () {54 vars in 912 bytes} [Sun Sep 25 00:56:00 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:56:00.297791 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e0f5cc-3292-46f4-ba9c-3e351d50236f None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:00.298107 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e0f5cc-3292-46f4-ba9c-3e351d50236f None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:00.298319 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e0f5cc-3292-46f4-ba9c-3e351d50236f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:00.336767 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-e5e0f5cc-3292-46f4-ba9c-3e351d50236f None None] MFA Rules not processed for user `20c274e96dec478b98ef51f80ea893b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:00.338047 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-e5e0f5cc-3292-46f4-ba9c-3e351d50236f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:00.380756 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae79333-8fbb-46d2-9381-15a6f0521299 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:00.382725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-2ae79333-8fbb-46d2-9381-15a6f0521299 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:00.385906 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 862/1725] 10.222.0.52 () {58 vars in 1009 bytes} [Sun Sep 25 00:56:00 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:00.414997 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae79333-8fbb-46d2-9381-15a6f0521299 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:00.427621 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283b9dfe-c570-433e-b1e3-fb1929744c19 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:00.440932 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283b9dfe-c570-433e-b1e3-fb1929744c19 service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:00.441454 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283b9dfe-c570-433e-b1e3-fb1929744c19 service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:00.441875 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283b9dfe-c570-433e-b1e3-fb1929744c19 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:00.473160 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-283b9dfe-c570-433e-b1e3-fb1929744c19 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:00.475414 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a16d2f60-55e3-4381-aa67-c39fc4af1023 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:00.475871 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a16d2f60-55e3-4381-aa67-c39fc4af1023 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:00.476275 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a16d2f60-55e3-4381-aa67-c39fc4af1023 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:00.495112 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-283b9dfe-c570-433e-b1e3-fb1929744c19 service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:00.539891 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 864/1726] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:56:00 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 159 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:56:00.560110 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-a16d2f60-55e3-4381-aa67-c39fc4af1023 None None] MFA Rules not processed for user `dbe551f8c10c4111b4ad7380a6482564`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:00.561382 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-a16d2f60-55e3-4381-aa67-c39fc4af1023 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:00.747620 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 863/1727] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:00 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 265 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 00:56:00.984546 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283b9dfe-c570-433e-b1e3-fb1929744c19 service neutron] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:00.987721 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-283b9dfe-c570-433e-b1e3-fb1929744c19 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:01.050780 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8e45d0-107d-43cf-a913-a85a8ae68dd1 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:01.051060 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8e45d0-107d-43cf-a913-a85a8ae68dd1 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:01.052098 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8e45d0-107d-43cf-a913-a85a8ae68dd1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:01.134759 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-db8e45d0-107d-43cf-a913-a85a8ae68dd1 None None] MFA Rules not processed for user `f6e6706abaea4d0797af19fdb0331ead`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:01.142670 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-db8e45d0-107d-43cf-a913-a85a8ae68dd1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:01.252946 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283b9dfe-c570-433e-b1e3-fb1929744c19 service neutron] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:01.261150 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dcb26b7-7914-46e3-a93b-28957530f0e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:01.262092 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dcb26b7-7914-46e3-a93b-28957530f0e5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:01.262595 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dcb26b7-7914-46e3-a93b-28957530f0e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:01.263064 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dcb26b7-7914-46e3-a93b-28957530f0e5 admin admin] PATH_INFO: `/v3/projects/67751478ae3a45f89000e9f561524d2c` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:01.264897 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dcb26b7-7914-46e3-a93b-28957530f0e5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=67751478ae3a45f89000e9f561524d2c)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:01.267464 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dcb26b7-7914-46e3-a93b-28957530f0e5 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:01.269732 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:01.390598 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 864/1728] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:01 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 341 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:01.392579 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db8e45d0-107d-43cf-a913-a85a8ae68dd1 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:01.422371 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-7dcb26b7-7914-46e3-a93b-28957530f0e5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '67751478ae3a45f89000e9f561524d2c', 'request_id': 'req-7dcb26b7-7914-46e3-a93b-28957530f0e5'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:01.431553 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db8e45d0-107d-43cf-a913-a85a8ae68dd1 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:01.436611 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-7dcb26b7-7914-46e3-a93b-28957530f0e5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '67751478ae3a45f89000e9f561524d2c', 'request_id': 'req-7dcb26b7-7914-46e3-a93b-28957530f0e5'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:01.454033 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-7dcb26b7-7914-46e3-a93b-28957530f0e5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '67751478ae3a45f89000e9f561524d2c', 'request_id': 'req-7dcb26b7-7914-46e3-a93b-28957530f0e5'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:01.473482 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-7dcb26b7-7914-46e3-a93b-28957530f0e5 admin admin] The token cache is being invalidate because project 67751478ae3a45f89000e9f561524d2c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:56:01.474102 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-7dcb26b7-7914-46e3-a93b-28957530f0e5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:01.474910 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-7dcb26b7-7914-46e3-a93b-28957530f0e5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:01.482118 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 865/1729] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:56:00 2022] DELETE /identity/v3/projects/67751478ae3a45f89000e9f561524d2c => generated 0 bytes in 493 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:01.483302 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dcb26b7-7914-46e3-a93b-28957530f0e5 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:01.493324 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-7dcb26b7-7914-46e3-a93b-28957530f0e5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:01.587648 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d65ba33-9b77-44fa-81fc-1274d20b3450 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f142988a4dd14a5f96eacea45c046f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:01.588479 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d65ba33-9b77-44fa-81fc-1274d20b3450 service nova] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:01.588907 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d65ba33-9b77-44fa-81fc-1274d20b3450 service nova] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:01.589301 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d65ba33-9b77-44fa-81fc-1274d20b3450 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:01.593374 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-4d65ba33-9b77-44fa-81fc-1274d20b3450 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:01.613538 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:01.682883 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dcb26b7-7914-46e3-a93b-28957530f0e5 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:01.723308 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-359f20a2-2607-403d-909b-7cba52125326 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '11a80b55dcd444df901c8bf80f2a2c26', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:01.724649 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359f20a2-2607-403d-909b-7cba52125326 service placement] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:01.725497 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359f20a2-2607-403d-909b-7cba52125326 service placement] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:01.725973 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359f20a2-2607-403d-909b-7cba52125326 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:01.729012 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-359f20a2-2607-403d-909b-7cba52125326 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:01.751394 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-359f20a2-2607-403d-909b-7cba52125326 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:01.753920 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-359f20a2-2607-403d-909b-7cba52125326 service placement] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:01.783454 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 866/1730] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:56:01 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 300 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:56:01.784289 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d65ba33-9b77-44fa-81fc-1274d20b3450 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:01.784863 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840083c0-aec7-4099-b99e-f8eccde5d08e None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:01.785073 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840083c0-aec7-4099-b99e-f8eccde5d08e None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:01.785373 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840083c0-aec7-4099-b99e-f8eccde5d08e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:01.786287 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d65ba33-9b77-44fa-81fc-1274d20b3450 service nova] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:01.808363 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 865/1731] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:56:01 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 416 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:56:01.810157 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01fe55d1-208a-4d93-a319-7d15fd8f882b None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:01.813344 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-840083c0-aec7-4099-b99e-f8eccde5d08e None None] MFA Rules not processed for user `f6e6706abaea4d0797af19fdb0331ead`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:01.815712 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01fe55d1-208a-4d93-a319-7d15fd8f882b None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:01.816119 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01fe55d1-208a-4d93-a319-7d15fd8f882b None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:01.816519 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-840083c0-aec7-4099-b99e-f8eccde5d08e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:01.822782 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 866/1732] 10.222.0.52 () {54 vars in 961 bytes} [Sun Sep 25 00:56:01 2022] GET /identity => generated 270 bytes in 14 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:56:01.829074 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee35597-d4c1-4556-af48-80fb55e80567 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:01.829407 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee35597-d4c1-4556-af48-80fb55e80567 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:01.829647 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee35597-d4c1-4556-af48-80fb55e80567 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:01.860852 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-8ee35597-d4c1-4556-af48-80fb55e80567 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:01.863391 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-8ee35597-d4c1-4556-af48-80fb55e80567 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:01.959042 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 867/1733] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:01 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:01.962926 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19653869-44e1-4c0d-8581-c2c0ace59a0f None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:01.963289 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19653869-44e1-4c0d-8581-c2c0ace59a0f None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:01.967174 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19653869-44e1-4c0d-8581-c2c0ace59a0f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:02.022566 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 867/1734] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:02.026898 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60312bf-9d10-4faf-9782-e74aa6a04d9a None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:02.027632 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60312bf-9d10-4faf-9782-e74aa6a04d9a None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:02.028561 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60312bf-9d10-4faf-9782-e74aa6a04d9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:02.038376 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-19653869-44e1-4c0d-8581-c2c0ace59a0f None None] MFA Rules not processed for user `8eabe2a74a604c47bc2e5c88b960dcb6`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:02.045865 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-19653869-44e1-4c0d-8581-c2c0ace59a0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:02.074399 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-e60312bf-9d10-4faf-9782-e74aa6a04d9a None None] MFA Rules not processed for user `11a80b55dcd444df901c8bf80f2a2c26`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:02.076821 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-e60312bf-9d10-4faf-9782-e74aa6a04d9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:02.105917 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:02.118246 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 868/1735] 10.222.0.52 () {58 vars in 1058 bytes} [Sun Sep 25 00:56:02 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:02.119467 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e60312bf-9d10-4faf-9782-e74aa6a04d9a None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:02.129393 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-e60312bf-9d10-4faf-9782-e74aa6a04d9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:02.166607 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e60312bf-9d10-4faf-9782-e74aa6a04d9a None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:02.184131 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c00969c-45ab-4f6e-9ab6-90a56f4d9f90 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f142988a4dd14a5f96eacea45c046f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:02.184779 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c00969c-45ab-4f6e-9ab6-90a56f4d9f90 service nova] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:02.185008 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c00969c-45ab-4f6e-9ab6-90a56f4d9f90 service nova] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:02.185320 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c00969c-45ab-4f6e-9ab6-90a56f4d9f90 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:02.195588 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-6c00969c-45ab-4f6e-9ab6-90a56f4d9f90 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:02.197768 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 868/1736] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:01 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 238 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:02.201055 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74dff193-9dc6-4df4-a3fd-107b95a5627a None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:02.201609 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74dff193-9dc6-4df4-a3fd-107b95a5627a None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:02.201978 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74dff193-9dc6-4df4-a3fd-107b95a5627a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:02.249155 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c00969c-45ab-4f6e-9ab6-90a56f4d9f90 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:02.250797 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c00969c-45ab-4f6e-9ab6-90a56f4d9f90 service nova] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:02.273019 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 869/1737] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:56:02 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:56:02.274552 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c00969c-45ab-4f6e-9ab6-90a56f4d9f90 service nova] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:02.296075 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-74dff193-9dc6-4df4-a3fd-107b95a5627a None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:02.298471 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c00969c-45ab-4f6e-9ab6-90a56f4d9f90 service nova] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:02.306725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-74dff193-9dc6-4df4-a3fd-107b95a5627a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:02.311829 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e54afb53-9f85-4e04-bc1a-15e34ee273aa service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '11a80b55dcd444df901c8bf80f2a2c26', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:02.313810 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54afb53-9f85-4e04-bc1a-15e34ee273aa service placement] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:02.315260 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54afb53-9f85-4e04-bc1a-15e34ee273aa service placement] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:02.317314 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54afb53-9f85-4e04-bc1a-15e34ee273aa service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:02.334450 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e54afb53-9f85-4e04-bc1a-15e34ee273aa service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:02.348842 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e54afb53-9f85-4e04-bc1a-15e34ee273aa service placement] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:02.370837 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 870/1738] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:56:02 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:56:02.371493 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 869/1739] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:02.375840 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e54afb53-9f85-4e04-bc1a-15e34ee273aa service placement] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:02.410565 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e54afb53-9f85-4e04-bc1a-15e34ee273aa service placement] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:02.420865 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c3e435-df4c-4cfb-8023-71e27869dc6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:02.421512 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c3e435-df4c-4cfb-8023-71e27869dc6a admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:02.421790 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c3e435-df4c-4cfb-8023-71e27869dc6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:02.422058 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c3e435-df4c-4cfb-8023-71e27869dc6a admin admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:02.422708 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c3e435-df4c-4cfb-8023-71e27869dc6a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:56:02.423137 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c3e435-df4c-4cfb-8023-71e27869dc6a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:02.424828 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c3e435-df4c-4cfb-8023-71e27869dc6a admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:02.466704 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 871/1740] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 00:56:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:56:02.470098 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74dff193-9dc6-4df4-a3fd-107b95a5627a None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:02.480838 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74dff193-9dc6-4df4-a3fd-107b95a5627a None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:02.495343 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45adfcb0-305c-49ef-a113-151c85b4f3dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:02.496365 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45adfcb0-305c-49ef-a113-151c85b4f3dc admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:02.496871 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45adfcb0-305c-49ef-a113-151c85b4f3dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:02.497340 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45adfcb0-305c-49ef-a113-151c85b4f3dc admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:02.499125 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45adfcb0-305c-49ef-a113-151c85b4f3dc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:02.501529 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45adfcb0-305c-49ef-a113-151c85b4f3dc admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:02.551526 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 870/1741] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:56:02 2022] POST /identity/v3/projects => generated 363 bytes in 82 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:56:02.556612 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c3e435-df4c-4cfb-8023-71e27869dc6a admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:02.598073 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c3e435-df4c-4cfb-8023-71e27869dc6a admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:02.703034 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc29683-c171-43d3-9141-98bc8c824576 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:02.704954 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc29683-c171-43d3-9141-98bc8c824576 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:02.705406 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc29683-c171-43d3-9141-98bc8c824576 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:02.705777 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc29683-c171-43d3-9141-98bc8c824576 admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:02.706637 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc29683-c171-43d3-9141-98bc8c824576 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:02.710502 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc29683-c171-43d3-9141-98bc8c824576 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:02.723623 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:02.761882 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 872/1742] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:56:02 2022] POST /identity/v3/users => generated 282 bytes in 207 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:56:02.766406 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45adfcb0-305c-49ef-a113-151c85b4f3dc admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:02.810933 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45adfcb0-305c-49ef-a113-151c85b4f3dc admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:02.827854 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f5ea769-fae3-499c-be32-8e129bb448ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:02.831093 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5ea769-fae3-499c-be32-8e129bb448ef admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:02.831641 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5ea769-fae3-499c-be32-8e129bb448ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:02.832383 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5ea769-fae3-499c-be32-8e129bb448ef admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:02.833847 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f5ea769-fae3-499c-be32-8e129bb448ef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:02.837779 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f5ea769-fae3-499c-be32-8e129bb448ef admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:02.891385 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 871/1743] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:02 2022] GET /identity/v3/roles => generated 1420 bytes in 126 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:56:02.895256 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc29683-c171-43d3-9141-98bc8c824576 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:02.916902 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc29683-c171-43d3-9141-98bc8c824576 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:02.926585 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-813eca1c-642a-4a2d-86bd-219129846b04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:02.927214 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813eca1c-642a-4a2d-86bd-219129846b04 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:02.927493 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813eca1c-642a-4a2d-86bd-219129846b04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:02.927764 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813eca1c-642a-4a2d-86bd-219129846b04 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:02.928585 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-813eca1c-642a-4a2d-86bd-219129846b04 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:02.935450 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-813eca1c-642a-4a2d-86bd-219129846b04 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:02.965439 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 873/1744] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:02 2022] GET /identity/v3/roles => generated 1420 bytes in 67 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:56:02.967036 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f5ea769-fae3-499c-be32-8e129bb448ef admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:02.983772 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f5ea769-fae3-499c-be32-8e129bb448ef admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:02.997527 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ceab317-a37d-4291-ac88-9c053529f602 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:02.998353 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ceab317-a37d-4291-ac88-9c053529f602 admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:02.998626 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ceab317-a37d-4291-ac88-9c053529f602 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:02.998894 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ceab317-a37d-4291-ac88-9c053529f602 admin admin] PATH_INFO: `/v3/projects/5ebd27a1c5574f37b514600466d568e2/users/5b8bc70fd9664d5594d6e63947fcde36/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:03.037586 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ceab317-a37d-4291-ac88-9c053529f602 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5ebd27a1c5574f37b514600466d568e2, user_id=5b8bc70fd9664d5594d6e63947fcde36, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:03.042953 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ceab317-a37d-4291-ac88-9c053529f602 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:03.107017 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 872/1745] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:56:02 2022] PUT /identity/v3/projects/5ebd27a1c5574f37b514600466d568e2/users/5b8bc70fd9664d5594d6e63947fcde36/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:03.121488 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00815ae6-6671-4852-ae3f-6aadf3ae9bf2 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:03.121900 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00815ae6-6671-4852-ae3f-6aadf3ae9bf2 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:03.122168 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00815ae6-6671-4852-ae3f-6aadf3ae9bf2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:03.190554 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-00815ae6-6671-4852-ae3f-6aadf3ae9bf2 None None] MFA Rules not processed for user `5b8bc70fd9664d5594d6e63947fcde36`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:03.193556 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-00815ae6-6671-4852-ae3f-6aadf3ae9bf2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:03.463551 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 874/1746] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:03 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 344 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 00:56:03.470914 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ceab317-a37d-4291-ac88-9c053529f602 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:03.668221 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ceab317-a37d-4291-ac88-9c053529f602 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:03.682656 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3391b1-7dae-45a1-9149-471e8cb826a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:03.683524 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3391b1-7dae-45a1-9149-471e8cb826a5 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:03.683896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3391b1-7dae-45a1-9149-471e8cb826a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:03.684258 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3391b1-7dae-45a1-9149-471e8cb826a5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:03.685057 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff3391b1-7dae-45a1-9149-471e8cb826a5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:03.687535 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff3391b1-7dae-45a1-9149-471e8cb826a5 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:03.742160 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 873/1747] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:56:03 2022] POST /identity/v3/projects => generated 365 bytes in 274 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:56:03.747261 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00815ae6-6671-4852-ae3f-6aadf3ae9bf2 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:03.764900 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00815ae6-6671-4852-ae3f-6aadf3ae9bf2 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:04.056818 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a657647-39e9-47d2-bfae-f8f46cfa6162 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:04.057479 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a657647-39e9-47d2-bfae-f8f46cfa6162 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:04.062768 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a657647-39e9-47d2-bfae-f8f46cfa6162 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:04.063164 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a657647-39e9-47d2-bfae-f8f46cfa6162 admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:04.063862 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a657647-39e9-47d2-bfae-f8f46cfa6162 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:04.075037 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a657647-39e9-47d2-bfae-f8f46cfa6162 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:04.159358 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 875/1748] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:56:03 2022] POST /identity/v3/users => generated 289 bytes in 413 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:56:04.165179 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3391b1-7dae-45a1-9149-471e8cb826a5 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:04.184521 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3391b1-7dae-45a1-9149-471e8cb826a5 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:04.201520 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a134e02-8989-4ca7-a58c-b504e21a291c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:04.202149 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a134e02-8989-4ca7-a58c-b504e21a291c admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:04.202425 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a134e02-8989-4ca7-a58c-b504e21a291c admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:04.204073 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a134e02-8989-4ca7-a58c-b504e21a291c admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:04.205801 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a134e02-8989-4ca7-a58c-b504e21a291c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:04.208600 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a134e02-8989-4ca7-a58c-b504e21a291c admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:04.225423 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 874/1749] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:04 2022] GET /identity/v3/roles => generated 1420 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:56:04.231015 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a657647-39e9-47d2-bfae-f8f46cfa6162 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:04.252908 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a657647-39e9-47d2-bfae-f8f46cfa6162 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:04.265095 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-365e01e4-f19b-455b-824d-bfa453c69458 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:04.267329 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365e01e4-f19b-455b-824d-bfa453c69458 admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:04.268631 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365e01e4-f19b-455b-824d-bfa453c69458 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:04.269062 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365e01e4-f19b-455b-824d-bfa453c69458 admin admin] PATH_INFO: `/v3/projects/b93a309e12d34f94b871601cc45e9255/users/915f2a9511d441dbb35110ad96fa918e/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:04.295957 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-365e01e4-f19b-455b-824d-bfa453c69458 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b93a309e12d34f94b871601cc45e9255, user_id=915f2a9511d441dbb35110ad96fa918e, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:04.307119 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-365e01e4-f19b-455b-824d-bfa453c69458 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:04.377658 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 876/1750] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:56:04 2022] PUT /identity/v3/projects/b93a309e12d34f94b871601cc45e9255/users/915f2a9511d441dbb35110ad96fa918e/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 00:56:04.384379 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a134e02-8989-4ca7-a58c-b504e21a291c admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:04.423861 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a134e02-8989-4ca7-a58c-b504e21a291c admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:04.578398 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:04.639598 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92f4bf21-65fb-44f6-869d-ace415955920 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:04.639746 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f4bf21-65fb-44f6-869d-ace415955920 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:04.640771 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f4bf21-65fb-44f6-869d-ace415955920 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:04.641061 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f4bf21-65fb-44f6-869d-ace415955920 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:04.645900 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-365e01e4-f19b-455b-824d-bfa453c69458 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:04.647875 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-365e01e4-f19b-455b-824d-bfa453c69458 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:04.653053 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92f4bf21-65fb-44f6-869d-ace415955920 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:04.678023 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92f4bf21-65fb-44f6-869d-ace415955920 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:04.720377 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-365e01e4-f19b-455b-824d-bfa453c69458 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:04.760870 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 875/1751] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:04 2022] GET /identity/v3/roles => generated 1420 bytes in 378 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:56:04.768382 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92f4bf21-65fb-44f6-869d-ace415955920 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:04.770617 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8c8a04-14af-4215-b8be-397c2a2444b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:04.776277 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8c8a04-14af-4215-b8be-397c2a2444b0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:04.777512 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8c8a04-14af-4215-b8be-397c2a2444b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:04.778011 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8c8a04-14af-4215-b8be-397c2a2444b0 admin admin] PATH_INFO: `/v3/users/a48e90a2614d4e098260868a34bb2af2` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:04.791967 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a8c8a04-14af-4215-b8be-397c2a2444b0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a48e90a2614d4e098260868a34bb2af2)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:04.806703 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a8c8a04-14af-4215-b8be-397c2a2444b0 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:04.829303 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92f4bf21-65fb-44f6-869d-ace415955920 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:04.876565 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ef3b3d5-ca09-4a16-b18c-da674469cff9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:04.877246 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef3b3d5-ca09-4a16-b18c-da674469cff9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:04.878219 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef3b3d5-ca09-4a16-b18c-da674469cff9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:04.878219 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef3b3d5-ca09-4a16-b18c-da674469cff9 admin admin] PATH_INFO: `/v3/projects/b93a309e12d34f94b871601cc45e9255/users/915f2a9511d441dbb35110ad96fa918e/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:04.904643 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ef3b3d5-ca09-4a16-b18c-da674469cff9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b93a309e12d34f94b871601cc45e9255, user_id=915f2a9511d441dbb35110ad96fa918e, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:04.908064 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ef3b3d5-ca09-4a16-b18c-da674469cff9 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:04.969830 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 876/1752] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:56:04 2022] PUT /identity/v3/projects/b93a309e12d34f94b871601cc45e9255/users/915f2a9511d441dbb35110ad96fa918e/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:04.982938 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:INSERT=6,keystone:SELECT=89,keystone:DELETE=5 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:04.984471 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce13197e-dba5-4ec6-b935-e94379e49a29 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:04.984754 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce13197e-dba5-4ec6-b935-e94379e49a29 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:04.985023 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce13197e-dba5-4ec6-b935-e94379e49a29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:05.028182 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-6a8c8a04-14af-4215-b8be-397c2a2444b0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a48e90a2614d4e098260868a34bb2af2'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:05.057652 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-6a8c8a04-14af-4215-b8be-397c2a2444b0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a48e90a2614d4e098260868a34bb2af2'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:05.059069 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-6a8c8a04-14af-4215-b8be-397c2a2444b0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a48e90a2614d4e098260868a34bb2af2'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:05.074080 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-ce13197e-dba5-4ec6-b935-e94379e49a29 None None] MFA Rules not processed for user `915f2a9511d441dbb35110ad96fa918e`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:05.075344 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-ce13197e-dba5-4ec6-b935-e94379e49a29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:05.079241 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-6a8c8a04-14af-4215-b8be-397c2a2444b0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a48e90a2614d4e098260868a34bb2af2'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:05.080379 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-6a8c8a04-14af-4215-b8be-397c2a2444b0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a48e90a2614d4e098260868a34bb2af2'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:05.131685 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 877/1753] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:56:04 2022] DELETE /identity/v3/users/a48e90a2614d4e098260868a34bb2af2 => generated 0 bytes in 486 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:05.140349 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4daf7db3-b3d0-40b7-87a9-16818a2e8a64 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:05.141197 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4daf7db3-b3d0-40b7-87a9-16818a2e8a64 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:05.141843 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4daf7db3-b3d0-40b7-87a9-16818a2e8a64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:05.189360 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-4daf7db3-b3d0-40b7-87a9-16818a2e8a64 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:05.192170 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-4daf7db3-b3d0-40b7-87a9-16818a2e8a64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:05.389036 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 878/1754] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 254 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:05.402358 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4daf7db3-b3d0-40b7-87a9-16818a2e8a64 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:05.409829 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-4daf7db3-b3d0-40b7-87a9-16818a2e8a64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:05.446328 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 877/1755] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:04 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 454 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:05.465346 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce13197e-dba5-4ec6-b935-e94379e49a29 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:05.467767 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-ce13197e-dba5-4ec6-b935-e94379e49a29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:05.675290 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4daf7db3-b3d0-40b7-87a9-16818a2e8a64 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:05.695908 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-797328b5-7c55-46c4-b09f-b3905ce3d128 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:05.699401 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797328b5-7c55-46c4-b09f-b3905ce3d128 service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:05.699823 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797328b5-7c55-46c4-b09f-b3905ce3d128 service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:05.700160 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797328b5-7c55-46c4-b09f-b3905ce3d128 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:05.713286 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce13197e-dba5-4ec6-b935-e94379e49a29 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:05.741403 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-797328b5-7c55-46c4-b09f-b3905ce3d128 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:05.749714 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-797328b5-7c55-46c4-b09f-b3905ce3d128 service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:05.796237 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 879/1756] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:56:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 395 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:56:05.865703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f8a1541-a66a-4c72-8fc7-b91871993b8a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:05.866332 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8a1541-a66a-4c72-8fc7-b91871993b8a service cinder] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:05.868118 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8a1541-a66a-4c72-8fc7-b91871993b8a service cinder] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:05.868371 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8a1541-a66a-4c72-8fc7-b91871993b8a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:05.915153 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:06.031480 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f8a1541-a66a-4c72-8fc7-b91871993b8a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:06.033456 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f8a1541-a66a-4c72-8fc7-b91871993b8a service cinder] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:06.073817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 878/1757] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:56:05 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 610 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 00:56:06.394345 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-797328b5-7c55-46c4-b09f-b3905ce3d128 service neutron] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:06.395787 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-797328b5-7c55-46c4-b09f-b3905ce3d128 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:06.426400 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-797328b5-7c55-46c4-b09f-b3905ce3d128 service neutron] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:06.459078 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5225e5ba-ada2-43a7-921e-8f98a9dad95d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:06.463013 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5225e5ba-ada2-43a7-921e-8f98a9dad95d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:06.463291 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5225e5ba-ada2-43a7-921e-8f98a9dad95d admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:06.466940 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5225e5ba-ada2-43a7-921e-8f98a9dad95d admin admin] PATH_INFO: `/v3/projects/03e39e5230b1454197fb977b74b18596` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:06.469894 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5225e5ba-ada2-43a7-921e-8f98a9dad95d admin admin] RBAC: Authorizing `identity:delete_project(project_id=03e39e5230b1454197fb977b74b18596)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:06.471974 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5225e5ba-ada2-43a7-921e-8f98a9dad95d admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:06.607166 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:06.611384 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-5225e5ba-ada2-43a7-921e-8f98a9dad95d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '03e39e5230b1454197fb977b74b18596', 'request_id': 'req-5225e5ba-ada2-43a7-921e-8f98a9dad95d'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:06.635411 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-5225e5ba-ada2-43a7-921e-8f98a9dad95d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '03e39e5230b1454197fb977b74b18596', 'request_id': 'req-5225e5ba-ada2-43a7-921e-8f98a9dad95d'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:06.643457 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-5225e5ba-ada2-43a7-921e-8f98a9dad95d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '03e39e5230b1454197fb977b74b18596', 'request_id': 'req-5225e5ba-ada2-43a7-921e-8f98a9dad95d'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:06.667275 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-5225e5ba-ada2-43a7-921e-8f98a9dad95d admin admin] The token cache is being invalidate because project 03e39e5230b1454197fb977b74b18596 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81386) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:56:06.667563 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-5225e5ba-ada2-43a7-921e-8f98a9dad95d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:06.669297 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-5225e5ba-ada2-43a7-921e-8f98a9dad95d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:06.674531 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 880/1758] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:56:06 2022] DELETE /identity/v3/projects/03e39e5230b1454197fb977b74b18596 => generated 0 bytes in 277 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 00:56:06.676600 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d664855-9642-4dac-8f53-ca224e116d94 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:06.677067 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d664855-9642-4dac-8f53-ca224e116d94 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:06.677481 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d664855-9642-4dac-8f53-ca224e116d94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:06.716541 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-0d664855-9642-4dac-8f53-ca224e116d94 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:06.718027 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-0d664855-9642-4dac-8f53-ca224e116d94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:06.817682 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5225e5ba-ada2-43a7-921e-8f98a9dad95d admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:06.819459 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-5225e5ba-ada2-43a7-921e-8f98a9dad95d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:06.884980 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 879/1759] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:06.893020 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f18b6a-9c9b-4b2b-be7a-d761f8aa697b None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:06.895575 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f18b6a-9c9b-4b2b-be7a-d761f8aa697b None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:06.897584 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f18b6a-9c9b-4b2b-be7a-d761f8aa697b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:06.979674 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-31f18b6a-9c9b-4b2b-be7a-d761f8aa697b None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:06.981368 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-31f18b6a-9c9b-4b2b-be7a-d761f8aa697b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:07.026582 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 880/1760] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 00:56:07.028896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f18b6a-9c9b-4b2b-be7a-d761f8aa697b None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:07.042087 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5225e5ba-ada2-43a7-921e-8f98a9dad95d admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:07.050732 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f18b6a-9c9b-4b2b-be7a-d761f8aa697b None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:07.055351 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1033ba57-5e5e-474e-8c01-e1d26e3b530d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:07.056393 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1033ba57-5e5e-474e-8c01-e1d26e3b530d service cinder] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:07.058391 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1033ba57-5e5e-474e-8c01-e1d26e3b530d service cinder] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:07.058834 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1033ba57-5e5e-474e-8c01-e1d26e3b530d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:07.062498 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f310e128-95c1-489a-99ab-02d14db84e42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:07.063245 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f310e128-95c1-489a-99ab-02d14db84e42 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:07.063767 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-1033ba57-5e5e-474e-8c01-e1d26e3b530d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:07.063917 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f310e128-95c1-489a-99ab-02d14db84e42 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:07.065566 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f310e128-95c1-489a-99ab-02d14db84e42 admin admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:07.067460 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f310e128-95c1-489a-99ab-02d14db84e42 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:56:07.067947 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f310e128-95c1-489a-99ab-02d14db84e42 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:07.072075 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f310e128-95c1-489a-99ab-02d14db84e42 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:07.099595 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 881/1761] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 00:56:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:56:07.111971 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f310e128-95c1-489a-99ab-02d14db84e42 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:07.151361 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f310e128-95c1-489a-99ab-02d14db84e42 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:07.168679 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f290676-89aa-44d2-b19f-8fbdcd10e87a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:07.171174 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f290676-89aa-44d2-b19f-8fbdcd10e87a admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:07.171692 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f290676-89aa-44d2-b19f-8fbdcd10e87a admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:07.172165 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f290676-89aa-44d2-b19f-8fbdcd10e87a admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:07.173077 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f290676-89aa-44d2-b19f-8fbdcd10e87a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:07.181171 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1033ba57-5e5e-474e-8c01-e1d26e3b530d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:07.184634 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1033ba57-5e5e-474e-8c01-e1d26e3b530d service cinder] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:07.188405 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f290676-89aa-44d2-b19f-8fbdcd10e87a admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:07.208265 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 881/1762] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:56:06 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 391 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 00:56:07.209697 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab53b0eb-cf1e-40b3-b608-98c212dd7293 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:07.209975 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab53b0eb-cf1e-40b3-b608-98c212dd7293 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:07.210246 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab53b0eb-cf1e-40b3-b608-98c212dd7293 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:07.259274 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-ab53b0eb-cf1e-40b3-b608-98c212dd7293 None None] MFA Rules not processed for user `dbe551f8c10c4111b4ad7380a6482564`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:07.260868 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-ab53b0eb-cf1e-40b3-b608-98c212dd7293 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:07.345590 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 882/1763] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:56:07 2022] POST /identity/v3/projects => generated 375 bytes in 234 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:56:07.351188 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f290676-89aa-44d2-b19f-8fbdcd10e87a admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:07.462804 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 882/1764] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:07 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 254 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:07.554234 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f8f8fe-8983-473c-bc94-a95cb8554f88 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:07.555237 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f8f8fe-8983-473c-bc94-a95cb8554f88 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:07.555724 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f8f8fe-8983-473c-bc94-a95cb8554f88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:07.641130 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f290676-89aa-44d2-b19f-8fbdcd10e87a admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:07.663535 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2e93a3-4053-451c-ab91-11bb2604349e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:07.665642 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2e93a3-4053-451c-ab91-11bb2604349e admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:07.667476 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2e93a3-4053-451c-ab91-11bb2604349e admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:07.668873 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2e93a3-4053-451c-ab91-11bb2604349e admin admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:07.672019 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee2e93a3-4053-451c-ab91-11bb2604349e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:07.688805 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee2e93a3-4053-451c-ab91-11bb2604349e admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:07.766616 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-67f8f8fe-8983-473c-bc94-a95cb8554f88 None None] MFA Rules not processed for user `3f20ffd8df994c1db1b1e7248db5b5a3`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:07.783398 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-67f8f8fe-8983-473c-bc94-a95cb8554f88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:07.795204 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 883/1765] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:56:07 2022] POST /identity/v3/users => generated 288 bytes in 445 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 00:56:07.798738 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:DELETE=4,keystone:SELECT=95,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:07.799084 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2e93a3-4053-451c-ab91-11bb2604349e admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:07.825500 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:07.842230 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2e93a3-4053-451c-ab91-11bb2604349e admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:07.866753 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e72492-54eb-4af9-a95a-4b0c03c51880 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:07.868726 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e72492-54eb-4af9-a95a-4b0c03c51880 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:07.868998 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e72492-54eb-4af9-a95a-4b0c03c51880 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:07.874940 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e72492-54eb-4af9-a95a-4b0c03c51880 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:07.877542 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79e72492-54eb-4af9-a95a-4b0c03c51880 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:07.879495 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79e72492-54eb-4af9-a95a-4b0c03c51880 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:07.925674 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 884/1766] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:07 2022] GET /identity/v3/roles => generated 1420 bytes in 130 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:56:07.935944 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e72492-54eb-4af9-a95a-4b0c03c51880 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:07.970470 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e72492-54eb-4af9-a95a-4b0c03c51880 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:07.985342 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a0d871-dd21-43be-bd5f-ac0f4eca6305 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:07.989654 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a0d871-dd21-43be-bd5f-ac0f4eca6305 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:07.991094 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a0d871-dd21-43be-bd5f-ac0f4eca6305 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:07.991383 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a0d871-dd21-43be-bd5f-ac0f4eca6305 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:08.010974 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 883/1767] 10.222.0.52 () {58 vars in 1009 bytes} [Sun Sep 25 00:56:07 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 458 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:08.014865 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2a0d871-dd21-43be-bd5f-ac0f4eca6305 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:08.022895 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2a0d871-dd21-43be-bd5f-ac0f4eca6305 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:08.077428 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 885/1768] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:07 2022] GET /identity/v3/roles => generated 1420 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:56:08.079988 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f8f8fe-8983-473c-bc94-a95cb8554f88 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:08.125868 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f8f8fe-8983-473c-bc94-a95cb8554f88 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:08.145827 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f3d99b-d319-4aaa-b7cb-b546e811fe4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:08.147711 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f3d99b-d319-4aaa-b7cb-b546e811fe4c admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:08.148978 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f3d99b-d319-4aaa-b7cb-b546e811fe4c admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:08.149819 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f3d99b-d319-4aaa-b7cb-b546e811fe4c admin admin] PATH_INFO: `/v3/projects/7cefb5d1dcb640a4b0ad87daa1418fae/users/09e222ed698c4972812c47408d2745cd/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:08.224611 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67f3d99b-d319-4aaa-b7cb-b546e811fe4c admin admin] RBAC: Authorizing `identity:create_grant(project_id=7cefb5d1dcb640a4b0ad87daa1418fae, user_id=09e222ed698c4972812c47408d2745cd, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:08.230572 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67f3d99b-d319-4aaa-b7cb-b546e811fe4c admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:08.243349 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a0d871-dd21-43be-bd5f-ac0f4eca6305 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:08.247171 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-d2a0d871-dd21-43be-bd5f-ac0f4eca6305 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:08.261360 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 884/1769] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:56:08 2022] PUT /identity/v3/projects/7cefb5d1dcb640a4b0ad87daa1418fae/users/09e222ed698c4972812c47408d2745cd/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:08.274351 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57561782-6a88-4426-9037-eb5ee10cd2ec None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:08.274815 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57561782-6a88-4426-9037-eb5ee10cd2ec None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:08.275207 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57561782-6a88-4426-9037-eb5ee10cd2ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:08.319677 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a0d871-dd21-43be-bd5f-ac0f4eca6305 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:08.379190 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-57561782-6a88-4426-9037-eb5ee10cd2ec None None] MFA Rules not processed for user `09e222ed698c4972812c47408d2745cd`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:08.382470 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-57561782-6a88-4426-9037-eb5ee10cd2ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:08.550133 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a402613-8450-43f1-af2b-43da0c9707cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:08.552051 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a402613-8450-43f1-af2b-43da0c9707cd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:08.552208 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a402613-8450-43f1-af2b-43da0c9707cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:08.553602 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a402613-8450-43f1-af2b-43da0c9707cd admin admin] PATH_INFO: `/v3/users/5b8bc70fd9664d5594d6e63947fcde36` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:08.558032 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a402613-8450-43f1-af2b-43da0c9707cd admin admin] RBAC: Authorizing `identity:delete_user(user_id=5b8bc70fd9664d5594d6e63947fcde36)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:08.563312 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a402613-8450-43f1-af2b-43da0c9707cd admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:08.662943 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 885/1770] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:08 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 390 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 00:56:08.664041 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57561782-6a88-4426-9037-eb5ee10cd2ec None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:08.668483 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-57561782-6a88-4426-9037-eb5ee10cd2ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:08.699885 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-6a402613-8450-43f1-af2b-43da0c9707cd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5b8bc70fd9664d5594d6e63947fcde36'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:08.702361 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-6a402613-8450-43f1-af2b-43da0c9707cd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5b8bc70fd9664d5594d6e63947fcde36'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:08.737107 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-6a402613-8450-43f1-af2b-43da0c9707cd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5b8bc70fd9664d5594d6e63947fcde36'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:08.738351 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-6a402613-8450-43f1-af2b-43da0c9707cd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5b8bc70fd9664d5594d6e63947fcde36'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:08.765094 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-6a402613-8450-43f1-af2b-43da0c9707cd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5b8bc70fd9664d5594d6e63947fcde36'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:08.851962 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 886/1771] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:56:08 2022] DELETE /identity/v3/users/5b8bc70fd9664d5594d6e63947fcde36 => generated 0 bytes in 610 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 00:56:08.854117 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a402613-8450-43f1-af2b-43da0c9707cd admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:08.859535 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-6a402613-8450-43f1-af2b-43da0c9707cd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:08.914786 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:08.962132 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57561782-6a88-4426-9037-eb5ee10cd2ec None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:09.063679 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a402613-8450-43f1-af2b-43da0c9707cd admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:09.096802 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c8bd3c-4e9d-4d01-a26f-f32b3a7dafe2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:09.097474 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c8bd3c-4e9d-4d01-a26f-f32b3a7dafe2 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:09.103714 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c8bd3c-4e9d-4d01-a26f-f32b3a7dafe2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:09.104103 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c8bd3c-4e9d-4d01-a26f-f32b3a7dafe2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:09.105093 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9c8bd3c-4e9d-4d01-a26f-f32b3a7dafe2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:09.114603 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9c8bd3c-4e9d-4d01-a26f-f32b3a7dafe2 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:09.144272 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:09.156100 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c85973fb-a942-4931-b3d4-43f4886ac72d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da606b3906dd4171bd76fd281e03e258', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:09.158109 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85973fb-a942-4931-b3d4-43f4886ac72d service swift] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:09.158825 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85973fb-a942-4931-b3d4-43f4886ac72d service swift] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:09.159225 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85973fb-a942-4931-b3d4-43f4886ac72d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:09.163374 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-c85973fb-a942-4931-b3d4-43f4886ac72d service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:09.240304 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 887/1772] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:56:08 2022] POST /identity/v3/projects => generated 375 bytes in 386 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:56:09.241803 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9420cd6d-5d5e-4ab3-aead-587fb554a755 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:09.242784 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9420cd6d-5d5e-4ab3-aead-587fb554a755 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:09.243108 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9420cd6d-5d5e-4ab3-aead-587fb554a755 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:09.322226 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-9420cd6d-5d5e-4ab3-aead-587fb554a755 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:09.323464 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-9420cd6d-5d5e-4ab3-aead-587fb554a755 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:09.397202 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c85973fb-a942-4931-b3d4-43f4886ac72d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:09.407958 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c85973fb-a942-4931-b3d4-43f4886ac72d service swift] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:09.437633 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 886/1773] 10.222.0.52 () {60 vars in 1384 bytes} [Sun Sep 25 00:56:08 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 774 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 25 00:56:09.438482 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c85973fb-a942-4931-b3d4-43f4886ac72d service swift] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:09.489681 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 888/1774] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 249 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:09.492832 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c85973fb-a942-4931-b3d4-43f4886ac72d service swift] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:09.499365 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9420cd6d-5d5e-4ab3-aead-587fb554a755 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:09.502885 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-9420cd6d-5d5e-4ab3-aead-587fb554a755 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:09.512512 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0aa72d1-f862-400e-88ac-965a2f10cec7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:09.514343 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0aa72d1-f862-400e-88ac-965a2f10cec7 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:09.515310 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0aa72d1-f862-400e-88ac-965a2f10cec7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:09.516063 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0aa72d1-f862-400e-88ac-965a2f10cec7 admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:09.518603 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0aa72d1-f862-400e-88ac-965a2f10cec7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:09.524564 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0aa72d1-f862-400e-88ac-965a2f10cec7 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:09.546916 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9420cd6d-5d5e-4ab3-aead-587fb554a755 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:09.574255 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 887/1775] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:56:09 2022] POST /identity/v3/users => generated 294 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:56:09.579007 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0aa72d1-f862-400e-88ac-965a2f10cec7 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:09.586671 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb9d82a0-a8e6-44bc-a4ff-e78dbc0b1b1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:09.593820 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9d82a0-a8e6-44bc-a4ff-e78dbc0b1b1b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:09.595138 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9d82a0-a8e6-44bc-a4ff-e78dbc0b1b1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:09.595751 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9d82a0-a8e6-44bc-a4ff-e78dbc0b1b1b admin admin] PATH_INFO: `/v3/users/bc68d01ffa694cab800a3944117fd768` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:09.600508 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0aa72d1-f862-400e-88ac-965a2f10cec7 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:09.602213 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb9d82a0-a8e6-44bc-a4ff-e78dbc0b1b1b admin admin] RBAC: Authorizing `identity:delete_user(user_id=bc68d01ffa694cab800a3944117fd768)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:09.609080 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb9d82a0-a8e6-44bc-a4ff-e78dbc0b1b1b admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:09.620495 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5efc8249-b187-47fc-9b61-a637fdca94c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:09.621215 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efc8249-b187-47fc-9b61-a637fdca94c8 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:09.623872 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efc8249-b187-47fc-9b61-a637fdca94c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:09.624147 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efc8249-b187-47fc-9b61-a637fdca94c8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:09.628846 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5efc8249-b187-47fc-9b61-a637fdca94c8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:09.633698 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5efc8249-b187-47fc-9b61-a637fdca94c8 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:09.688612 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 888/1776] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:09 2022] GET /identity/v3/roles => generated 1420 bytes in 108 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:56:09.688612 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5efc8249-b187-47fc-9b61-a637fdca94c8 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:09.697649 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-5efc8249-b187-47fc-9b61-a637fdca94c8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:09.833074 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-eb9d82a0-a8e6-44bc-a4ff-e78dbc0b1b1b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bc68d01ffa694cab800a3944117fd768'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:09.839657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-eb9d82a0-a8e6-44bc-a4ff-e78dbc0b1b1b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bc68d01ffa694cab800a3944117fd768'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:09.864941 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-eb9d82a0-a8e6-44bc-a4ff-e78dbc0b1b1b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bc68d01ffa694cab800a3944117fd768'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:09.868820 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-eb9d82a0-a8e6-44bc-a4ff-e78dbc0b1b1b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bc68d01ffa694cab800a3944117fd768'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:09.896844 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-eb9d82a0-a8e6-44bc-a4ff-e78dbc0b1b1b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bc68d01ffa694cab800a3944117fd768'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:09.944834 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 889/1777] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:56:09 2022] DELETE /identity/v3/users/bc68d01ffa694cab800a3944117fd768 => generated 0 bytes in 454 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:09.948080 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb9d82a0-a8e6-44bc-a4ff-e78dbc0b1b1b admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:09.951646 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-eb9d82a0-a8e6-44bc-a4ff-e78dbc0b1b1b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:09.976313 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5efc8249-b187-47fc-9b61-a637fdca94c8 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:10.125823 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:10.278544 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:10.283220 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb9d82a0-a8e6-44bc-a4ff-e78dbc0b1b1b admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:10.295040 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ee3d35-e93f-481c-a8a3-110703c5bb1f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:10.296034 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ee3d35-e93f-481c-a8a3-110703c5bb1f service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:10.296654 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ee3d35-e93f-481c-a8a3-110703c5bb1f service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:10.299092 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a935ca-32c5-4703-8304-bb9e742893d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:10.299587 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ee3d35-e93f-481c-a8a3-110703c5bb1f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:10.309277 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a935ca-32c5-4703-8304-bb9e742893d4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:10.310589 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a935ca-32c5-4703-8304-bb9e742893d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:10.315352 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-01ee3d35-e93f-481c-a8a3-110703c5bb1f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:10.318428 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a935ca-32c5-4703-8304-bb9e742893d4 admin admin] PATH_INFO: `/v3/projects/ebcbfdd554ab41299a6932ab451d0072/users/8c48b627d65c470f83b259720f83b106/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:10.396304 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01ee3d35-e93f-481c-a8a3-110703c5bb1f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:10.397949 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01ee3d35-e93f-481c-a8a3-110703c5bb1f service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:10.412181 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78a935ca-32c5-4703-8304-bb9e742893d4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ebcbfdd554ab41299a6932ab451d0072, user_id=8c48b627d65c470f83b259720f83b106, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:10.415728 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78a935ca-32c5-4703-8304-bb9e742893d4 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:10.433815 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 889/1778] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:56:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 744 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:56:10.436173 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1955426e-987b-4307-89d0-f3ca6472ac5b None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:10.438027 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1955426e-987b-4307-89d0-f3ca6472ac5b None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:10.438661 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1955426e-987b-4307-89d0-f3ca6472ac5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:10.451244 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 890/1779] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:56:09 2022] PUT /identity/v3/projects/ebcbfdd554ab41299a6932ab451d0072/users/8c48b627d65c470f83b259720f83b106/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 506 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:10.455107 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a935ca-32c5-4703-8304-bb9e742893d4 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:10.506227 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-1955426e-987b-4307-89d0-f3ca6472ac5b None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:10.508753 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-1955426e-987b-4307-89d0-f3ca6472ac5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:10.675900 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a935ca-32c5-4703-8304-bb9e742893d4 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:10.683338 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e78df9d4-a455-4148-80ac-af612482039d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:10.684208 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78df9d4-a455-4148-80ac-af612482039d admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:10.684531 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78df9d4-a455-4148-80ac-af612482039d admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:10.693711 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78df9d4-a455-4148-80ac-af612482039d admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:10.696268 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e78df9d4-a455-4148-80ac-af612482039d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:10.699366 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e78df9d4-a455-4148-80ac-af612482039d admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:10.738352 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 891/1780] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:10 2022] GET /identity/v3/roles => generated 1420 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:56:10.742002 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e78df9d4-a455-4148-80ac-af612482039d admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:10.800781 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e78df9d4-a455-4148-80ac-af612482039d admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:10.828603 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b09179f-f14b-4ee7-94ad-03a23248e704 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:10.829280 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b09179f-f14b-4ee7-94ad-03a23248e704 admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:10.829560 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b09179f-f14b-4ee7-94ad-03a23248e704 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:10.829835 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b09179f-f14b-4ee7-94ad-03a23248e704 admin admin] PATH_INFO: `/v3/projects/ebcbfdd554ab41299a6932ab451d0072/users/8c48b627d65c470f83b259720f83b106/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:10.851311 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b09179f-f14b-4ee7-94ad-03a23248e704 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ebcbfdd554ab41299a6932ab451d0072, user_id=8c48b627d65c470f83b259720f83b106, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:10.863581 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 890/1781] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 429 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:10.865324 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b09179f-f14b-4ee7-94ad-03a23248e704 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:10.906837 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 892/1782] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:56:10 2022] PUT /identity/v3/projects/ebcbfdd554ab41299a6932ab451d0072/users/8c48b627d65c470f83b259720f83b106/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:10.922686 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3f63ee-30a0-4b9d-83b7-8311a931b46e None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:10.922965 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3f63ee-30a0-4b9d-83b7-8311a931b46e None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:10.927508 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3f63ee-30a0-4b9d-83b7-8311a931b46e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:10.994545 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b09179f-f14b-4ee7-94ad-03a23248e704 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:10.996646 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-8b09179f-f14b-4ee7-94ad-03a23248e704 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:11.040057 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-5d3f63ee-30a0-4b9d-83b7-8311a931b46e None None] MFA Rules not processed for user `8c48b627d65c470f83b259720f83b106`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:11.047123 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-5d3f63ee-30a0-4b9d-83b7-8311a931b46e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:11.219635 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b09179f-f14b-4ee7-94ad-03a23248e704 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:11.240789 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76cc96de-1be4-4839-8f0d-bef5bd2d4b5d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:11.241335 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76cc96de-1be4-4839-8f0d-bef5bd2d4b5d service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:11.241604 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76cc96de-1be4-4839-8f0d-bef5bd2d4b5d service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:11.241929 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76cc96de-1be4-4839-8f0d-bef5bd2d4b5d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:11.437879 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 891/1783] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:10 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 515 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 00:56:11.448191 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d3f63ee-30a0-4b9d-83b7-8311a931b46e None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:11.460458 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-5d3f63ee-30a0-4b9d-83b7-8311a931b46e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:11.530866 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76cc96de-1be4-4839-8f0d-bef5bd2d4b5d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:11.540189 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76cc96de-1be4-4839-8f0d-bef5bd2d4b5d service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:11.591681 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 893/1784] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:56:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 597 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:56:11.685072 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d3f63ee-30a0-4b9d-83b7-8311a931b46e None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:11.705576 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d93746e-85bb-4ee8-b10f-fab2cc0fa161 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:11.706209 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d93746e-85bb-4ee8-b10f-fab2cc0fa161 service cinder] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:11.706480 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d93746e-85bb-4ee8-b10f-fab2cc0fa161 service cinder] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:11.706744 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d93746e-85bb-4ee8-b10f-fab2cc0fa161 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:11.723263 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-3d93746e-85bb-4ee8-b10f-fab2cc0fa161 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:11.751576 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:12.056663 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d93746e-85bb-4ee8-b10f-fab2cc0fa161 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:12.056663 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d93746e-85bb-4ee8-b10f-fab2cc0fa161 service cinder] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:12.131201 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 892/1785] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:56:11 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 679 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:56:12.188765 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76cc96de-1be4-4839-8f0d-bef5bd2d4b5d service neutron] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:12.190395 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-76cc96de-1be4-4839-8f0d-bef5bd2d4b5d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:12.222893 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76cc96de-1be4-4839-8f0d-bef5bd2d4b5d service neutron] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:12.235673 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75225b0a-9301-43d0-91a9-e80a20599d19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:12.236637 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75225b0a-9301-43d0-91a9-e80a20599d19 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:12.237172 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75225b0a-9301-43d0-91a9-e80a20599d19 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:12.239672 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75225b0a-9301-43d0-91a9-e80a20599d19 admin admin] PATH_INFO: `/v3/users/f105f660be9c42a895d6fd19262884cf` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:12.244531 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75225b0a-9301-43d0-91a9-e80a20599d19 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f105f660be9c42a895d6fd19262884cf)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:12.253316 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75225b0a-9301-43d0-91a9-e80a20599d19 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:12.269208 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:12.383569 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-75225b0a-9301-43d0-91a9-e80a20599d19 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f105f660be9c42a895d6fd19262884cf'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:12.384265 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-75225b0a-9301-43d0-91a9-e80a20599d19 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f105f660be9c42a895d6fd19262884cf'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:12.399971 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-75225b0a-9301-43d0-91a9-e80a20599d19 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f105f660be9c42a895d6fd19262884cf'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:12.400929 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-75225b0a-9301-43d0-91a9-e80a20599d19 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f105f660be9c42a895d6fd19262884cf'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:12.419373 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-75225b0a-9301-43d0-91a9-e80a20599d19 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f105f660be9c42a895d6fd19262884cf'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:12.464507 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 894/1786] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:56:12 2022] DELETE /identity/v3/users/f105f660be9c42a895d6fd19262884cf => generated 0 bytes in 276 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:12.469963 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c10d341-8935-4d11-9092-0d2b054527de None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:12.470402 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c10d341-8935-4d11-9092-0d2b054527de None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:12.470667 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c10d341-8935-4d11-9092-0d2b054527de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:12.524914 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-2c10d341-8935-4d11-9092-0d2b054527de None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:12.528430 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-2c10d341-8935-4d11-9092-0d2b054527de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:12.925722 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 893/1787] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 457 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:13.030916 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75225b0a-9301-43d0-91a9-e80a20599d19 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:13.033667 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-75225b0a-9301-43d0-91a9-e80a20599d19 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:13.119218 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc04f69-196b-4e05-b4e9-1c58121c0f70 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:13.119841 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc04f69-196b-4e05-b4e9-1c58121c0f70 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:13.120275 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc04f69-196b-4e05-b4e9-1c58121c0f70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:13.192064 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-8cc04f69-196b-4e05-b4e9-1c58121c0f70 None None] MFA Rules not processed for user `20c274e96dec478b98ef51f80ea893b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:13.197529 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-8cc04f69-196b-4e05-b4e9-1c58121c0f70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:13.223621 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 894/1788] 10.222.0.57 () {58 vars in 1015 bytes} [Sun Sep 25 00:56:13 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:13.237597 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75225b0a-9301-43d0-91a9-e80a20599d19 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:13.261861 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-913c5ce6-5318-476f-8448-70d1408e3872 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:13.262425 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-913c5ce6-5318-476f-8448-70d1408e3872 service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:13.262698 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-913c5ce6-5318-476f-8448-70d1408e3872 service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:13.262961 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-913c5ce6-5318-476f-8448-70d1408e3872 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:13.334393 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-913c5ce6-5318-476f-8448-70d1408e3872 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:13.337325 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-913c5ce6-5318-476f-8448-70d1408e3872 service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:13.364720 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 895/1789] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:56:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 335 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:56:13.715689 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc04f69-196b-4e05-b4e9-1c58121c0f70 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:13.731903 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc04f69-196b-4e05-b4e9-1c58121c0f70 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:13.744093 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a40649-21d2-4a3f-bffc-0235e59f3008 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:13.744736 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a40649-21d2-4a3f-bffc-0235e59f3008 service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:13.745166 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a40649-21d2-4a3f-bffc-0235e59f3008 service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:13.745563 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a40649-21d2-4a3f-bffc-0235e59f3008 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:13.775363 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a40649-21d2-4a3f-bffc-0235e59f3008 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:13.786209 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a40649-21d2-4a3f-bffc-0235e59f3008 service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:13.821425 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 895/1790] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:56:13 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:56:14.659040 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-913c5ce6-5318-476f-8448-70d1408e3872 service neutron] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:14.661625 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-913c5ce6-5318-476f-8448-70d1408e3872 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:14.707238 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-913c5ce6-5318-476f-8448-70d1408e3872 service neutron] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:14.743921 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b669b8c-29a3-4c88-8966-a5b272404272 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:14.744577 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b669b8c-29a3-4c88-8966-a5b272404272 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:14.744865 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b669b8c-29a3-4c88-8966-a5b272404272 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:14.755230 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b669b8c-29a3-4c88-8966-a5b272404272 admin admin] PATH_INFO: `/v3/projects/5ebd27a1c5574f37b514600466d568e2` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:14.760090 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b669b8c-29a3-4c88-8966-a5b272404272 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5ebd27a1c5574f37b514600466d568e2)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:14.761730 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b669b8c-29a3-4c88-8966-a5b272404272 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:15.017502 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-5b669b8c-29a3-4c88-8966-a5b272404272 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5ebd27a1c5574f37b514600466d568e2', 'request_id': 'req-5b669b8c-29a3-4c88-8966-a5b272404272'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:15.047297 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-5b669b8c-29a3-4c88-8966-a5b272404272 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5ebd27a1c5574f37b514600466d568e2', 'request_id': 'req-5b669b8c-29a3-4c88-8966-a5b272404272'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:15.093635 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-5b669b8c-29a3-4c88-8966-a5b272404272 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5ebd27a1c5574f37b514600466d568e2', 'request_id': 'req-5b669b8c-29a3-4c88-8966-a5b272404272'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:15.141425 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-5b669b8c-29a3-4c88-8966-a5b272404272 admin admin] The token cache is being invalidate because project 5ebd27a1c5574f37b514600466d568e2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:56:15.141816 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-5b669b8c-29a3-4c88-8966-a5b272404272 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:15.149133 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-5b669b8c-29a3-4c88-8966-a5b272404272 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:15.159035 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 896/1791] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:56:14 2022] DELETE /identity/v3/projects/5ebd27a1c5574f37b514600466d568e2 => generated 0 bytes in 500 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:15.163323 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a40649-21d2-4a3f-bffc-0235e59f3008 service neutron] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:15.191771 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a40649-21d2-4a3f-bffc-0235e59f3008 service neutron] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:15.378507 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b669b8c-29a3-4c88-8966-a5b272404272 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:15.380322 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-5b669b8c-29a3-4c88-8966-a5b272404272 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:15.442167 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:15.456208 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fddf47ae-f5a5-4dd8-946e-f6a88ba92ba2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:15.457087 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddf47ae-f5a5-4dd8-946e-f6a88ba92ba2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:15.457530 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddf47ae-f5a5-4dd8-946e-f6a88ba92ba2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:15.461123 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddf47ae-f5a5-4dd8-946e-f6a88ba92ba2 admin admin] PATH_INFO: `/v3/users/915f2a9511d441dbb35110ad96fa918e` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:15.469286 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fddf47ae-f5a5-4dd8-946e-f6a88ba92ba2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=915f2a9511d441dbb35110ad96fa918e)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:15.488955 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fddf47ae-f5a5-4dd8-946e-f6a88ba92ba2 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:15.606321 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:15.657723 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-fddf47ae-f5a5-4dd8-946e-f6a88ba92ba2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '915f2a9511d441dbb35110ad96fa918e'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:15.671455 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b669b8c-29a3-4c88-8966-a5b272404272 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:15.680795 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-fddf47ae-f5a5-4dd8-946e-f6a88ba92ba2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '915f2a9511d441dbb35110ad96fa918e'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:15.683982 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-fddf47ae-f5a5-4dd8-946e-f6a88ba92ba2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '915f2a9511d441dbb35110ad96fa918e'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:15.685907 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b09a915-5786-42bc-823c-9c6242a36085 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:15.686772 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b09a915-5786-42bc-823c-9c6242a36085 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:15.688921 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b09a915-5786-42bc-823c-9c6242a36085 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:15.689470 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b09a915-5786-42bc-823c-9c6242a36085 admin admin] PATH_INFO: `/v3/projects/122985cea56d4df18d25bf3e04482c9a` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:15.694087 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b09a915-5786-42bc-823c-9c6242a36085 admin admin] RBAC: Authorizing `identity:delete_project(project_id=122985cea56d4df18d25bf3e04482c9a)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:15.700011 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b09a915-5786-42bc-823c-9c6242a36085 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:15.704642 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-fddf47ae-f5a5-4dd8-946e-f6a88ba92ba2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '915f2a9511d441dbb35110ad96fa918e'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:15.707422 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-fddf47ae-f5a5-4dd8-946e-f6a88ba92ba2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '915f2a9511d441dbb35110ad96fa918e'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:15.754281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 896/1792] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:56:15 2022] DELETE /identity/v3/users/915f2a9511d441dbb35110ad96fa918e => generated 0 bytes in 588 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 00:56:15.811214 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-5b09a915-5786-42bc-823c-9c6242a36085 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '122985cea56d4df18d25bf3e04482c9a', 'request_id': 'req-5b09a915-5786-42bc-823c-9c6242a36085'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:15.823233 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-5b09a915-5786-42bc-823c-9c6242a36085 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '122985cea56d4df18d25bf3e04482c9a', 'request_id': 'req-5b09a915-5786-42bc-823c-9c6242a36085'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:15.832284 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-5b09a915-5786-42bc-823c-9c6242a36085 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '122985cea56d4df18d25bf3e04482c9a', 'request_id': 'req-5b09a915-5786-42bc-823c-9c6242a36085'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:15.851582 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-5b09a915-5786-42bc-823c-9c6242a36085 admin admin] The token cache is being invalidate because project 122985cea56d4df18d25bf3e04482c9a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:56:15.851870 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-5b09a915-5786-42bc-823c-9c6242a36085 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:15.852694 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-5b09a915-5786-42bc-823c-9c6242a36085 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:15.857082 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 897/1793] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:56:15 2022] DELETE /identity/v3/projects/122985cea56d4df18d25bf3e04482c9a => generated 0 bytes in 477 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:15.858808 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fddf47ae-f5a5-4dd8-946e-f6a88ba92ba2 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:15.861410 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-fddf47ae-f5a5-4dd8-946e-f6a88ba92ba2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:15.992017 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fddf47ae-f5a5-4dd8-946e-f6a88ba92ba2 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:16.111920 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1f6e77-ba99-4c81-a70c-866c1b225c54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:16.112754 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1f6e77-ba99-4c81-a70c-866c1b225c54 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:16.113455 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1f6e77-ba99-4c81-a70c-866c1b225c54 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:16.113872 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1f6e77-ba99-4c81-a70c-866c1b225c54 admin admin] PATH_INFO: `/v3/users/1e45257a614e4018b47f7292178979c9` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:16.115240 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1f6e77-ba99-4c81-a70c-866c1b225c54 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1e45257a614e4018b47f7292178979c9)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:16.120463 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1f6e77-ba99-4c81-a70c-866c1b225c54 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:16.253589 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-ed1f6e77-ba99-4c81-a70c-866c1b225c54 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1e45257a614e4018b47f7292178979c9'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:16.271279 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-ed1f6e77-ba99-4c81-a70c-866c1b225c54 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1e45257a614e4018b47f7292178979c9'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:16.277852 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-ed1f6e77-ba99-4c81-a70c-866c1b225c54 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1e45257a614e4018b47f7292178979c9'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:16.279656 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:16.300818 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-ed1f6e77-ba99-4c81-a70c-866c1b225c54 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1e45257a614e4018b47f7292178979c9'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:16.303595 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-ed1f6e77-ba99-4c81-a70c-866c1b225c54 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1e45257a614e4018b47f7292178979c9'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:16.339351 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 897/1794] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:56:15 2022] DELETE /identity/v3/users/1e45257a614e4018b47f7292178979c9 => generated 0 bytes in 481 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:16.567545 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b09a915-5786-42bc-823c-9c6242a36085 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:16.569441 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-5b09a915-5786-42bc-823c-9c6242a36085 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:16.606344 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b09a915-5786-42bc-823c-9c6242a36085 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:16.716907 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1f6e77-ba99-4c81-a70c-866c1b225c54 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:16.721828 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-ed1f6e77-ba99-4c81-a70c-866c1b225c54 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:16.730986 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe4bb81-ea7d-4ede-b9e9-cbdc03dfd4ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:16.732193 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe4bb81-ea7d-4ede-b9e9-cbdc03dfd4ca admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:16.732615 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe4bb81-ea7d-4ede-b9e9-cbdc03dfd4ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:16.732956 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe4bb81-ea7d-4ede-b9e9-cbdc03dfd4ca admin admin] PATH_INFO: `/v3/projects/55b9b458194a4f4084ea2186c20b1efd` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:16.734840 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fe4bb81-ea7d-4ede-b9e9-cbdc03dfd4ca admin admin] RBAC: Authorizing `identity:delete_project(project_id=55b9b458194a4f4084ea2186c20b1efd)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:16.739934 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fe4bb81-ea7d-4ede-b9e9-cbdc03dfd4ca admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:16.747065 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1f6e77-ba99-4c81-a70c-866c1b225c54 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:16.756170 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c53e6f17-6f02-481e-9c64-1eb2c8003ba0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:16.757953 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c53e6f17-6f02-481e-9c64-1eb2c8003ba0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:16.759960 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c53e6f17-6f02-481e-9c64-1eb2c8003ba0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:16.761117 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c53e6f17-6f02-481e-9c64-1eb2c8003ba0 admin admin] PATH_INFO: `/v3/projects/b93a309e12d34f94b871601cc45e9255` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:16.763376 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c53e6f17-6f02-481e-9c64-1eb2c8003ba0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b93a309e12d34f94b871601cc45e9255)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:16.766678 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c53e6f17-6f02-481e-9c64-1eb2c8003ba0 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:16.824847 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:16.834192 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-9fe4bb81-ea7d-4ede-b9e9-cbdc03dfd4ca admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '55b9b458194a4f4084ea2186c20b1efd', 'request_id': 'req-9fe4bb81-ea7d-4ede-b9e9-cbdc03dfd4ca'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:16.849336 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-9fe4bb81-ea7d-4ede-b9e9-cbdc03dfd4ca admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '55b9b458194a4f4084ea2186c20b1efd', 'request_id': 'req-9fe4bb81-ea7d-4ede-b9e9-cbdc03dfd4ca'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:16.857933 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-9fe4bb81-ea7d-4ede-b9e9-cbdc03dfd4ca admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '55b9b458194a4f4084ea2186c20b1efd', 'request_id': 'req-9fe4bb81-ea7d-4ede-b9e9-cbdc03dfd4ca'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:16.876705 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-9fe4bb81-ea7d-4ede-b9e9-cbdc03dfd4ca admin admin] The token cache is being invalidate because project 55b9b458194a4f4084ea2186c20b1efd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:56:16.877167 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-9fe4bb81-ea7d-4ede-b9e9-cbdc03dfd4ca admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:16.877920 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-9fe4bb81-ea7d-4ede-b9e9-cbdc03dfd4ca admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:16.883372 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 898/1795] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:56:16 2022] DELETE /identity/v3/projects/55b9b458194a4f4084ea2186c20b1efd => generated 0 bytes in 313 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 00:56:16.885344 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe4bb81-ea7d-4ede-b9e9-cbdc03dfd4ca admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:16.886785 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-9fe4bb81-ea7d-4ede-b9e9-cbdc03dfd4ca admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:16.897153 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-c53e6f17-6f02-481e-9c64-1eb2c8003ba0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b93a309e12d34f94b871601cc45e9255', 'request_id': 'req-c53e6f17-6f02-481e-9c64-1eb2c8003ba0'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:16.909429 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-c53e6f17-6f02-481e-9c64-1eb2c8003ba0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b93a309e12d34f94b871601cc45e9255', 'request_id': 'req-c53e6f17-6f02-481e-9c64-1eb2c8003ba0'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:16.919658 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-c53e6f17-6f02-481e-9c64-1eb2c8003ba0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b93a309e12d34f94b871601cc45e9255', 'request_id': 'req-c53e6f17-6f02-481e-9c64-1eb2c8003ba0'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:16.937306 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-c53e6f17-6f02-481e-9c64-1eb2c8003ba0 admin admin] The token cache is being invalidate because project b93a309e12d34f94b871601cc45e9255 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81386) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:56:16.937858 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-c53e6f17-6f02-481e-9c64-1eb2c8003ba0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:16.938852 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-c53e6f17-6f02-481e-9c64-1eb2c8003ba0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:16.940794 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 898/1796] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:56:16 2022] DELETE /identity/v3/projects/b93a309e12d34f94b871601cc45e9255 => generated 0 bytes in 226 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:16.942372 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c8481c-89f1-4288-ab56-e5a4303fe9eb None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:16.943524 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c8481c-89f1-4288-ab56-e5a4303fe9eb None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:16.944241 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c8481c-89f1-4288-ab56-e5a4303fe9eb None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:16.946695 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 899/1797] 10.222.0.52 () {54 vars in 921 bytes} [Sun Sep 25 00:56:16 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:56:16.947983 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09753896-552b-43a5-8cc9-4a10546d7ed0 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:16.948571 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09753896-552b-43a5-8cc9-4a10546d7ed0 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:16.949006 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09753896-552b-43a5-8cc9-4a10546d7ed0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:16.992518 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-09753896-552b-43a5-8cc9-4a10546d7ed0 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:16.999081 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-09753896-552b-43a5-8cc9-4a10546d7ed0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:17.042837 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe4bb81-ea7d-4ede-b9e9-cbdc03dfd4ca admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:17.171863 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 900/1798] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 224 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:17.173000 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f550dce-8f16-4e2b-8456-2e138e02be77 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:17.173343 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f550dce-8f16-4e2b-8456-2e138e02be77 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:17.173799 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f550dce-8f16-4e2b-8456-2e138e02be77 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:17.183718 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c12ded10-5764-4139-b70d-29c8f7620e61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:17.184266 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12ded10-5764-4139-b70d-29c8f7620e61 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:17.184541 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12ded10-5764-4139-b70d-29c8f7620e61 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:17.187586 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12ded10-5764-4139-b70d-29c8f7620e61 admin admin] PATH_INFO: `/v3/users/5aad523f24fa4be5a28b25a4a1bbe56f` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:17.189210 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c12ded10-5764-4139-b70d-29c8f7620e61 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5aad523f24fa4be5a28b25a4a1bbe56f)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:17.191215 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c12ded10-5764-4139-b70d-29c8f7620e61 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:17.199686 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-7f550dce-8f16-4e2b-8456-2e138e02be77 None None] MFA Rules not processed for user `f142988a4dd14a5f96eacea45c046f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:17.201415 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-7f550dce-8f16-4e2b-8456-2e138e02be77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:17.317766 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:17.344963 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-c12ded10-5764-4139-b70d-29c8f7620e61 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5aad523f24fa4be5a28b25a4a1bbe56f'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:17.348818 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-c12ded10-5764-4139-b70d-29c8f7620e61 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5aad523f24fa4be5a28b25a4a1bbe56f'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:17.362345 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-c12ded10-5764-4139-b70d-29c8f7620e61 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5aad523f24fa4be5a28b25a4a1bbe56f'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:17.364826 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-c12ded10-5764-4139-b70d-29c8f7620e61 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5aad523f24fa4be5a28b25a4a1bbe56f'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:17.380486 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 901/1799] 10.222.0.52 () {58 vars in 1018 bytes} [Sun Sep 25 00:56:17 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:17.382112 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-c12ded10-5764-4139-b70d-29c8f7620e61 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5aad523f24fa4be5a28b25a4a1bbe56f'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:17.383018 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07ceb08-ce31-4ad7-adfc-528b050e3de4 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:17.383470 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07ceb08-ce31-4ad7-adfc-528b050e3de4 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:17.383886 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07ceb08-ce31-4ad7-adfc-528b050e3de4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:17.408981 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:17.421106 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 899/1800] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:56:16 2022] DELETE /identity/v3/users/5aad523f24fa4be5a28b25a4a1bbe56f => generated 0 bytes in 537 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:17.421975 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c12ded10-5764-4139-b70d-29c8f7620e61 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:17.422161 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-a07ceb08-ce31-4ad7-adfc-528b050e3de4 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:17.423859 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-a07ceb08-ce31-4ad7-adfc-528b050e3de4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:17.424178 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-c12ded10-5764-4139-b70d-29c8f7620e61 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:17.575025 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c12ded10-5764-4139-b70d-29c8f7620e61 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:17.593676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71fe0767-b1fa-44d4-84c1-c3735ef1844d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f142988a4dd14a5f96eacea45c046f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:17.594354 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71fe0767-b1fa-44d4-84c1-c3735ef1844d service nova] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:17.594637 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71fe0767-b1fa-44d4-84c1-c3735ef1844d service nova] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:17.594908 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71fe0767-b1fa-44d4-84c1-c3735ef1844d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:17.614303 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71fe0767-b1fa-44d4-84c1-c3735ef1844d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:17.620322 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 902/1801] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 239 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:17.621552 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07ceb08-ce31-4ad7-adfc-528b050e3de4 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:17.622380 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71fe0767-b1fa-44d4-84c1-c3735ef1844d service nova] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:17.622845 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-a07ceb08-ce31-4ad7-adfc-528b050e3de4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:17.641078 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07ceb08-ce31-4ad7-adfc-528b050e3de4 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:17.646248 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d9bffa3-e5af-4dcf-a510-c01ac6989359 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:17.647229 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9bffa3-e5af-4dcf-a510-c01ac6989359 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:17.647349 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 900/1802] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:56:17 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 226 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:56:17.648257 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71fe0767-b1fa-44d4-84c1-c3735ef1844d service nova] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:17.658869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71fe0767-b1fa-44d4-84c1-c3735ef1844d service nova] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:17.659414 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9bffa3-e5af-4dcf-a510-c01ac6989359 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:17.659932 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9bffa3-e5af-4dcf-a510-c01ac6989359 admin admin] PATH_INFO: `/v3/projects/c549d8eee4e64e0196bb387d08150c6a` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:17.663712 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b48c1c6-2353-4b1f-b374-0d0126e303ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:17.664388 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b48c1c6-2353-4b1f-b374-0d0126e303ed admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:17.664788 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b48c1c6-2353-4b1f-b374-0d0126e303ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:17.665060 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b48c1c6-2353-4b1f-b374-0d0126e303ed admin admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:17.665437 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d9bffa3-e5af-4dcf-a510-c01ac6989359 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c549d8eee4e64e0196bb387d08150c6a)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:17.666144 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b48c1c6-2353-4b1f-b374-0d0126e303ed admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:56:17.666595 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b48c1c6-2353-4b1f-b374-0d0126e303ed admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:17.668052 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d9bffa3-e5af-4dcf-a510-c01ac6989359 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:17.668845 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b48c1c6-2353-4b1f-b374-0d0126e303ed admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:17.686228 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 901/1803] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 00:56:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:56:17.687130 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b48c1c6-2353-4b1f-b374-0d0126e303ed admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:17.698717 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b48c1c6-2353-4b1f-b374-0d0126e303ed admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:17.704134 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e607a821-ce20-40b1-9516-e9a06cae2fe3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:17.704740 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e607a821-ce20-40b1-9516-e9a06cae2fe3 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:17.705024 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e607a821-ce20-40b1-9516-e9a06cae2fe3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:17.705280 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e607a821-ce20-40b1-9516-e9a06cae2fe3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:17.705931 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e607a821-ce20-40b1-9516-e9a06cae2fe3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:17.707888 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e607a821-ce20-40b1-9516-e9a06cae2fe3 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:17.761511 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 902/1804] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:56:17 2022] POST /identity/v3/projects => generated 369 bytes in 70 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 00:56:17.763059 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e607a821-ce20-40b1-9516-e9a06cae2fe3 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:17.790200 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-2d9bffa3-e5af-4dcf-a510-c01ac6989359 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c549d8eee4e64e0196bb387d08150c6a', 'request_id': 'req-2d9bffa3-e5af-4dcf-a510-c01ac6989359'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:17.805379 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-2d9bffa3-e5af-4dcf-a510-c01ac6989359 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c549d8eee4e64e0196bb387d08150c6a', 'request_id': 'req-2d9bffa3-e5af-4dcf-a510-c01ac6989359'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:17.818896 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-2d9bffa3-e5af-4dcf-a510-c01ac6989359 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c549d8eee4e64e0196bb387d08150c6a', 'request_id': 'req-2d9bffa3-e5af-4dcf-a510-c01ac6989359'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:17.834804 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-2d9bffa3-e5af-4dcf-a510-c01ac6989359 admin admin] The token cache is being invalidate because project c549d8eee4e64e0196bb387d08150c6a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81386) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:56:17.835361 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-2d9bffa3-e5af-4dcf-a510-c01ac6989359 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:17.836780 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-2d9bffa3-e5af-4dcf-a510-c01ac6989359 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:17.839023 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 903/1805] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:56:17 2022] DELETE /identity/v3/projects/c549d8eee4e64e0196bb387d08150c6a => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:17.844262 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8969b5c9-431d-4d12-b5e4-f6173612a1d2 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:17.844616 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8969b5c9-431d-4d12-b5e4-f6173612a1d2 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:17.845080 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8969b5c9-431d-4d12-b5e4-f6173612a1d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:17.871829 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-8969b5c9-431d-4d12-b5e4-f6173612a1d2 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:17.873276 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-8969b5c9-431d-4d12-b5e4-f6173612a1d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:17.899025 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e607a821-ce20-40b1-9516-e9a06cae2fe3 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:17.908840 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3e95475-3a4e-4551-a342-fff728fa5c85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:17.909646 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e95475-3a4e-4551-a342-fff728fa5c85 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:17.909920 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e95475-3a4e-4551-a342-fff728fa5c85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:17.910192 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e95475-3a4e-4551-a342-fff728fa5c85 admin admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:17.910855 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3e95475-3a4e-4551-a342-fff728fa5c85 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:17.913700 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3e95475-3a4e-4551-a342-fff728fa5c85 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:17.944137 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 903/1806] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:56:17 2022] POST /identity/v3/users => generated 285 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:56:17.944137 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3e95475-3a4e-4551-a342-fff728fa5c85 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:17.944472 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-c3e95475-3a4e-4551-a342-fff728fa5c85 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:17.956794 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 904/1807] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 00:56:17.957848 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8969b5c9-431d-4d12-b5e4-f6173612a1d2 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:17.958186 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3e95475-3a4e-4551-a342-fff728fa5c85 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:17.964516 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb860f5-d6eb-47ce-b80b-a0f11627460a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:17.965187 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb860f5-d6eb-47ce-b80b-a0f11627460a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:17.965482 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb860f5-d6eb-47ce-b80b-a0f11627460a admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:17.965752 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb860f5-d6eb-47ce-b80b-a0f11627460a admin admin] PATH_INFO: `/v3/projects/8928bfafbc184624805c19c6ecd97656` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:17.966169 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-8969b5c9-431d-4d12-b5e4-f6173612a1d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:17.966969 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fb860f5-d6eb-47ce-b80b-a0f11627460a admin admin] RBAC: Authorizing `identity:delete_project(project_id=8928bfafbc184624805c19c6ecd97656)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:17.968842 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fb860f5-d6eb-47ce-b80b-a0f11627460a admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:17.978520 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8969b5c9-431d-4d12-b5e4-f6173612a1d2 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:17.983368 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85f311a5-8ccc-4d46-9e37-7fbdbb18133d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:17.983971 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f311a5-8ccc-4d46-9e37-7fbdbb18133d admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:17.984246 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f311a5-8ccc-4d46-9e37-7fbdbb18133d admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:17.984514 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f311a5-8ccc-4d46-9e37-7fbdbb18133d admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:17.985190 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85f311a5-8ccc-4d46-9e37-7fbdbb18133d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:17.987682 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85f311a5-8ccc-4d46-9e37-7fbdbb18133d admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:17.995822 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:18.001714 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 905/1808] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:17 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:56:18.003750 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55624297-baef-4bb2-9fd1-c55de1d68886 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:18.004445 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55624297-baef-4bb2-9fd1-c55de1d68886 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:18.005449 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=94,keystone:INSERT=4,keystone:DELETE=2 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:18.006339 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55624297-baef-4bb2-9fd1-c55de1d68886 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:18.021368 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-7fb860f5-d6eb-47ce-b80b-a0f11627460a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8928bfafbc184624805c19c6ecd97656', 'request_id': 'req-7fb860f5-d6eb-47ce-b80b-a0f11627460a'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:18.031294 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-7fb860f5-d6eb-47ce-b80b-a0f11627460a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8928bfafbc184624805c19c6ecd97656', 'request_id': 'req-7fb860f5-d6eb-47ce-b80b-a0f11627460a'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:18.032520 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-55624297-baef-4bb2-9fd1-c55de1d68886 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:18.034870 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-55624297-baef-4bb2-9fd1-c55de1d68886 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:18.039032 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-7fb860f5-d6eb-47ce-b80b-a0f11627460a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8928bfafbc184624805c19c6ecd97656', 'request_id': 'req-7fb860f5-d6eb-47ce-b80b-a0f11627460a'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:18.055109 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-7fb860f5-d6eb-47ce-b80b-a0f11627460a admin admin] The token cache is being invalidate because project 8928bfafbc184624805c19c6ecd97656 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:56:18.055672 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-7fb860f5-d6eb-47ce-b80b-a0f11627460a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:18.056652 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-7fb860f5-d6eb-47ce-b80b-a0f11627460a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:18.058627 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 904/1809] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:56:17 2022] DELETE /identity/v3/projects/8928bfafbc184624805c19c6ecd97656 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:18.062785 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb860f5-d6eb-47ce-b80b-a0f11627460a admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:18.064950 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-7fb860f5-d6eb-47ce-b80b-a0f11627460a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:18.138423 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 906/1810] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:18.140245 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55624297-baef-4bb2-9fd1-c55de1d68886 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:18.142080 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-55624297-baef-4bb2-9fd1-c55de1d68886 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:18.156941 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55624297-baef-4bb2-9fd1-c55de1d68886 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:18.163435 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e0ba23-84e0-4866-b063-4d2fca354491 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:18.164700 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e0ba23-84e0-4866-b063-4d2fca354491 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:18.165126 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e0ba23-84e0-4866-b063-4d2fca354491 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:18.165583 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e0ba23-84e0-4866-b063-4d2fca354491 admin admin] PATH_INFO: `/v3/users/8078163880b6454cbe8b6bce54b52ad0` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:18.167104 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e0ba23-84e0-4866-b063-4d2fca354491 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8078163880b6454cbe8b6bce54b52ad0)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:18.169806 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e0ba23-84e0-4866-b063-4d2fca354491 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:18.174141 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb860f5-d6eb-47ce-b80b-a0f11627460a admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:18.191797 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeb055d7-7ded-47e7-ab47-33cf292d9fa4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:18.192766 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb055d7-7ded-47e7-ab47-33cf292d9fa4 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:18.193319 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb055d7-7ded-47e7-ab47-33cf292d9fa4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:18.193793 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb055d7-7ded-47e7-ab47-33cf292d9fa4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:18.195673 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeb055d7-7ded-47e7-ab47-33cf292d9fa4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:18.206429 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeb055d7-7ded-47e7-ab47-33cf292d9fa4 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:18.243190 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 905/1811] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:18 2022] GET /identity/v3/roles => generated 1420 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:56:18.244084 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeb055d7-7ded-47e7-ab47-33cf292d9fa4 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:18.246589 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-c9e0ba23-84e0-4866-b063-4d2fca354491 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8078163880b6454cbe8b6bce54b52ad0'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:18.260423 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeb055d7-7ded-47e7-ab47-33cf292d9fa4 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:18.262247 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-c9e0ba23-84e0-4866-b063-4d2fca354491 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8078163880b6454cbe8b6bce54b52ad0'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:18.263207 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-c9e0ba23-84e0-4866-b063-4d2fca354491 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8078163880b6454cbe8b6bce54b52ad0'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:18.265854 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4e7990-f103-4fe3-9aba-6d7f1fd4c33a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:18.266529 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4e7990-f103-4fe3-9aba-6d7f1fd4c33a admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:18.266810 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4e7990-f103-4fe3-9aba-6d7f1fd4c33a admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:18.267119 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4e7990-f103-4fe3-9aba-6d7f1fd4c33a admin admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:18.267768 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e4e7990-f103-4fe3-9aba-6d7f1fd4c33a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:56:18.268077 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e4e7990-f103-4fe3-9aba-6d7f1fd4c33a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:18.276950 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-c9e0ba23-84e0-4866-b063-4d2fca354491 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8078163880b6454cbe8b6bce54b52ad0'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:18.277916 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-c9e0ba23-84e0-4866-b063-4d2fca354491 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8078163880b6454cbe8b6bce54b52ad0'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:18.278991 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e4e7990-f103-4fe3-9aba-6d7f1fd4c33a admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:18.304964 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 906/1812] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 00:56:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:56:18.304964 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4e7990-f103-4fe3-9aba-6d7f1fd4c33a admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:18.307070 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-5e4e7990-f103-4fe3-9aba-6d7f1fd4c33a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:18.315729 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 907/1813] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:56:18 2022] DELETE /identity/v3/users/8078163880b6454cbe8b6bce54b52ad0 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:18.317010 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e0ba23-84e0-4866-b063-4d2fca354491 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:18.330869 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-c9e0ba23-84e0-4866-b063-4d2fca354491 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:18.339060 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4e7990-f103-4fe3-9aba-6d7f1fd4c33a admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:18.515823 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-785b0f54-fcd9-4bb2-b8ce-72876a222022 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:18.516755 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785b0f54-fcd9-4bb2-b8ce-72876a222022 admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:18.517039 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785b0f54-fcd9-4bb2-b8ce-72876a222022 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:18.517401 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e0ba23-84e0-4866-b063-4d2fca354491 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:18.517518 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785b0f54-fcd9-4bb2-b8ce-72876a222022 admin admin] PATH_INFO: `/v3/projects/48f37c93452a40fb9f1070965497a755/users/800fd23f307c470a849d3a92f1c3c831/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:18.524080 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8635cd-8742-45ca-b297-6cc0dd542a29 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:18.525080 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8635cd-8742-45ca-b297-6cc0dd542a29 service cinder] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:18.525391 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8635cd-8742-45ca-b297-6cc0dd542a29 service cinder] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:18.525666 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8635cd-8742-45ca-b297-6cc0dd542a29 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:18.528240 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-5a8635cd-8742-45ca-b297-6cc0dd542a29 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:18.561319 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-785b0f54-fcd9-4bb2-b8ce-72876a222022 admin admin] RBAC: Authorizing `identity:create_grant(project_id=48f37c93452a40fb9f1070965497a755, user_id=800fd23f307c470a849d3a92f1c3c831, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:18.561319 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-785b0f54-fcd9-4bb2-b8ce-72876a222022 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:18.572614 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:18.590334 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 907/1814] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:56:18 2022] PUT /identity/v3/projects/48f37c93452a40fb9f1070965497a755/users/800fd23f307c470a849d3a92f1c3c831/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 288 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:18.591398 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-785b0f54-fcd9-4bb2-b8ce-72876a222022 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:18.603287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-785b0f54-fcd9-4bb2-b8ce-72876a222022 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:18.680154 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a8635cd-8742-45ca-b297-6cc0dd542a29 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:18.681812 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a8635cd-8742-45ca-b297-6cc0dd542a29 service cinder] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:18.687071 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=93,keystone:DELETE=5,keystone:INSERT=2 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:18.706990 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 908/1815] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:56:18 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 390 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:56:18.713001 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba829dc-ec40-464c-b84c-b9104d0834ab None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:18.713321 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba829dc-ec40-464c-b84c-b9104d0834ab None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:18.713574 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba829dc-ec40-464c-b84c-b9104d0834ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:18.774670 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-2ba829dc-ec40-464c-b84c-b9104d0834ab None None] MFA Rules not processed for user `800fd23f307c470a849d3a92f1c3c831`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:18.775984 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-2ba829dc-ec40-464c-b84c-b9104d0834ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:18.791614 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-785b0f54-fcd9-4bb2-b8ce-72876a222022 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:18.807963 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-317c2284-7f06-4561-bbcf-f79f0a9aecb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:18.808604 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317c2284-7f06-4561-bbcf-f79f0a9aecb6 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:18.808901 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317c2284-7f06-4561-bbcf-f79f0a9aecb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:18.809443 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317c2284-7f06-4561-bbcf-f79f0a9aecb6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:18.810208 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-317c2284-7f06-4561-bbcf-f79f0a9aecb6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:18.816622 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-317c2284-7f06-4561-bbcf-f79f0a9aecb6 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:18.906289 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 908/1816] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:56:18 2022] POST /identity/v3/projects => generated 409 bytes in 315 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:56:18.906712 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 909/1817] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:18 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:18.914840 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ba829dc-ec40-464c-b84c-b9104d0834ab None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:18.915045 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-317c2284-7f06-4561-bbcf-f79f0a9aecb6 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:18.941515 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ba829dc-ec40-464c-b84c-b9104d0834ab None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:19.149658 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b521d25c-6d04-4eb2-8c76-9df246a61c12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:19.152488 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b521d25c-6d04-4eb2-8c76-9df246a61c12 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:19.152776 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b521d25c-6d04-4eb2-8c76-9df246a61c12 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:19.153058 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b521d25c-6d04-4eb2-8c76-9df246a61c12 admin admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:19.157537 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b521d25c-6d04-4eb2-8c76-9df246a61c12 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:19.162066 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b521d25c-6d04-4eb2-8c76-9df246a61c12 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:19.177335 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-317c2284-7f06-4561-bbcf-f79f0a9aecb6 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:19.192567 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07ff820-00e1-4952-8a95-b42af29ddd28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:19.193579 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07ff820-00e1-4952-8a95-b42af29ddd28 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:19.193936 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07ff820-00e1-4952-8a95-b42af29ddd28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:19.194336 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07ff820-00e1-4952-8a95-b42af29ddd28 admin admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:19.195222 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a07ff820-00e1-4952-8a95-b42af29ddd28 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:19.217814 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a07ff820-00e1-4952-8a95-b42af29ddd28 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:19.272334 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 909/1818] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:56:18 2022] POST /identity/v3/users => generated 305 bytes in 358 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:56:19.275478 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07ff820-00e1-4952-8a95-b42af29ddd28 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:19.277242 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-a07ff820-00e1-4952-8a95-b42af29ddd28 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:19.293421 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07ff820-00e1-4952-8a95-b42af29ddd28 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:19.301620 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 910/1819] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:56:18 2022] POST /identity/v3/projects => generated 369 bytes in 384 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:56:19.301620 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b521d25c-6d04-4eb2-8c76-9df246a61c12 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:19.425906 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:19.441288 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d67f8f-32b0-4c3f-8043-a74f5f1f1c9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:19.444558 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d67f8f-32b0-4c3f-8043-a74f5f1f1c9e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:19.445581 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d67f8f-32b0-4c3f-8043-a74f5f1f1c9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:19.445865 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d67f8f-32b0-4c3f-8043-a74f5f1f1c9e admin admin] PATH_INFO: `/v3/projects/a0aa105c58b1474cbe337ec83f25fdfc` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:19.449693 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d67f8f-32b0-4c3f-8043-a74f5f1f1c9e admin admin] RBAC: Authorizing `identity:delete_project(project_id=a0aa105c58b1474cbe337ec83f25fdfc)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:19.451798 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d67f8f-32b0-4c3f-8043-a74f5f1f1c9e admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:19.484464 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b521d25c-6d04-4eb2-8c76-9df246a61c12 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:19.495114 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22fe8322-37b9-4207-b4b6-17cdeb3600cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:19.496222 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22fe8322-37b9-4207-b4b6-17cdeb3600cb admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:19.496776 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22fe8322-37b9-4207-b4b6-17cdeb3600cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:19.497767 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22fe8322-37b9-4207-b4b6-17cdeb3600cb admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:19.498876 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22fe8322-37b9-4207-b4b6-17cdeb3600cb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:19.501030 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22fe8322-37b9-4207-b4b6-17cdeb3600cb admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:19.605372 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 911/1820] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:19 2022] GET /identity/v3/roles => generated 1420 bytes in 306 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:56:19.606047 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=93,keystone:INSERT=4,keystone:DELETE=3 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:19.607196 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22fe8322-37b9-4207-b4b6-17cdeb3600cb admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:19.630226 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22fe8322-37b9-4207-b4b6-17cdeb3600cb admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:19.647994 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-c2d67f8f-32b0-4c3f-8043-a74f5f1f1c9e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a0aa105c58b1474cbe337ec83f25fdfc', 'request_id': 'req-c2d67f8f-32b0-4c3f-8043-a74f5f1f1c9e'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:19.669891 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-c2d67f8f-32b0-4c3f-8043-a74f5f1f1c9e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a0aa105c58b1474cbe337ec83f25fdfc', 'request_id': 'req-c2d67f8f-32b0-4c3f-8043-a74f5f1f1c9e'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:19.713677 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-c2d67f8f-32b0-4c3f-8043-a74f5f1f1c9e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a0aa105c58b1474cbe337ec83f25fdfc', 'request_id': 'req-c2d67f8f-32b0-4c3f-8043-a74f5f1f1c9e'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:19.763528 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-c2d67f8f-32b0-4c3f-8043-a74f5f1f1c9e admin admin] The token cache is being invalidate because project a0aa105c58b1474cbe337ec83f25fdfc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:56:19.764241 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-c2d67f8f-32b0-4c3f-8043-a74f5f1f1c9e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:19.765822 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-c2d67f8f-32b0-4c3f-8043-a74f5f1f1c9e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:19.769772 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 910/1821] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:56:19 2022] DELETE /identity/v3/projects/a0aa105c58b1474cbe337ec83f25fdfc => generated 0 bytes in 497 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:19.770918 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d67f8f-32b0-4c3f-8043-a74f5f1f1c9e admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:19.777995 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-c2d67f8f-32b0-4c3f-8043-a74f5f1f1c9e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:19.803078 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a317a95-6f39-4838-bec8-80c8704f210c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:19.803913 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a317a95-6f39-4838-bec8-80c8704f210c admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:19.804250 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a317a95-6f39-4838-bec8-80c8704f210c admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:19.804656 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a317a95-6f39-4838-bec8-80c8704f210c admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:19.805448 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a317a95-6f39-4838-bec8-80c8704f210c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:19.808533 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a317a95-6f39-4838-bec8-80c8704f210c admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:19.845993 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 912/1822] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:56:19 2022] POST /identity/v3/users => generated 291 bytes in 240 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:56:19.853953 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2e03e1-1bbe-4127-8293-4a32a46cc557 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:19.854270 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2e03e1-1bbe-4127-8293-4a32a46cc557 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:19.854540 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2e03e1-1bbe-4127-8293-4a32a46cc557 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:19.908191 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d67f8f-32b0-4c3f-8043-a74f5f1f1c9e admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:19.911297 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-ca2e03e1-1bbe-4127-8293-4a32a46cc557 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:19.912794 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-ca2e03e1-1bbe-4127-8293-4a32a46cc557 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:19.917169 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06963fb1-4647-489d-8d06-f5fc028e4ac2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:19.918858 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06963fb1-4647-489d-8d06-f5fc028e4ac2 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:19.919448 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06963fb1-4647-489d-8d06-f5fc028e4ac2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:19.920042 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06963fb1-4647-489d-8d06-f5fc028e4ac2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:19.921611 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06963fb1-4647-489d-8d06-f5fc028e4ac2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:19.923808 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 913/1823] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 71 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:19.924801 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06963fb1-4647-489d-8d06-f5fc028e4ac2 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:19.925424 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2e03e1-1bbe-4127-8293-4a32a46cc557 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:19.932168 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-ca2e03e1-1bbe-4127-8293-4a32a46cc557 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:19.950975 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 911/1824] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:19 2022] GET /identity/v3/roles => generated 1420 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:56:19.953005 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95edcc8-4977-479c-9af6-d4082f0d383e None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:19.953586 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95edcc8-4977-479c-9af6-d4082f0d383e None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:19.954182 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95edcc8-4977-479c-9af6-d4082f0d383e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:19.993685 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2e03e1-1bbe-4127-8293-4a32a46cc557 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:19.995322 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-c95edcc8-4977-479c-9af6-d4082f0d383e None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:19.998798 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-c95edcc8-4977-479c-9af6-d4082f0d383e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:20.013284 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b3d97fb-ac0e-4316-9b75-73ddde20facf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:20.014354 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3d97fb-ac0e-4316-9b75-73ddde20facf admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:20.014758 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3d97fb-ac0e-4316-9b75-73ddde20facf admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:20.015053 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3d97fb-ac0e-4316-9b75-73ddde20facf admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:20.015812 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b3d97fb-ac0e-4316-9b75-73ddde20facf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:20.025380 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 912/1825] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 70 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:20.025380 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c95edcc8-4977-479c-9af6-d4082f0d383e None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:20.035039 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b3d97fb-ac0e-4316-9b75-73ddde20facf admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:20.041461 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c95edcc8-4977-479c-9af6-d4082f0d383e None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:20.072555 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1354ee-a9bb-4936-af6a-c0d4c3909637 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:20.073446 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1354ee-a9bb-4936-af6a-c0d4c3909637 admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:20.073806 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1354ee-a9bb-4936-af6a-c0d4c3909637 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:20.074215 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1354ee-a9bb-4936-af6a-c0d4c3909637 admin admin] PATH_INFO: `/v3/projects/f33e879b885c49fda61a093fdebbd9cd/users/c9a5d45e4e6d4d38813d0884d3fbf1f1/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:20.079649 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 914/1826] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:19 2022] GET /identity/v3/roles => generated 1420 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:56:20.079649 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b3d97fb-ac0e-4316-9b75-73ddde20facf admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:20.116531 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b3d97fb-ac0e-4316-9b75-73ddde20facf admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:20.123772 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9808137d-531c-4df7-baee-21ab2afcffed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:20.125440 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9808137d-531c-4df7-baee-21ab2afcffed admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:20.126278 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9808137d-531c-4df7-baee-21ab2afcffed admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:20.127057 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9808137d-531c-4df7-baee-21ab2afcffed admin admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:20.128933 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9808137d-531c-4df7-baee-21ab2afcffed admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:56:20.129506 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9808137d-531c-4df7-baee-21ab2afcffed admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:20.133491 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9808137d-531c-4df7-baee-21ab2afcffed admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:20.138274 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c1354ee-a9bb-4936-af6a-c0d4c3909637 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f33e879b885c49fda61a093fdebbd9cd, user_id=c9a5d45e4e6d4d38813d0884d3fbf1f1, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:20.146248 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c1354ee-a9bb-4936-af6a-c0d4c3909637 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:20.162951 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 915/1827] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 00:56:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:56:20.165470 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9808137d-531c-4df7-baee-21ab2afcffed admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:20.180304 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9808137d-531c-4df7-baee-21ab2afcffed admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:20.190794 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 913/1828] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:56:20 2022] PUT /identity/v3/projects/f33e879b885c49fda61a093fdebbd9cd/users/c9a5d45e4e6d4d38813d0884d3fbf1f1/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:20.201474 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1354ee-a9bb-4936-af6a-c0d4c3909637 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:20.353896 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-949f9241-e3c7-4d05-94a7-b387d1c0056b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:20.355437 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-949f9241-e3c7-4d05-94a7-b387d1c0056b admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:20.355813 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-949f9241-e3c7-4d05-94a7-b387d1c0056b admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:20.356615 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-949f9241-e3c7-4d05-94a7-b387d1c0056b admin admin] PATH_INFO: `/v3/projects/0ba384f2387f43539de482688bcccf66/users/d062f5ba56b347efab90e047e77e74e7/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:20.371406 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1354ee-a9bb-4936-af6a-c0d4c3909637 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:20.396879 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7272661d-7fae-4005-8454-7e1f0cf18a63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:20.397912 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7272661d-7fae-4005-8454-7e1f0cf18a63 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:20.398641 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-949f9241-e3c7-4d05-94a7-b387d1c0056b admin admin] RBAC: Authorizing `identity:create_grant(project_id=0ba384f2387f43539de482688bcccf66, user_id=d062f5ba56b347efab90e047e77e74e7, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:20.399096 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7272661d-7fae-4005-8454-7e1f0cf18a63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:20.399380 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7272661d-7fae-4005-8454-7e1f0cf18a63 admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:20.400044 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7272661d-7fae-4005-8454-7e1f0cf18a63 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:20.402006 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7272661d-7fae-4005-8454-7e1f0cf18a63 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:20.402226 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-949f9241-e3c7-4d05-94a7-b387d1c0056b admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:20.437576 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 916/1829] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:56:20 2022] PUT /identity/v3/projects/0ba384f2387f43539de482688bcccf66/users/d062f5ba56b347efab90e047e77e74e7/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 274 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:20.444254 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c565147f-c12c-4a4b-bd86-b17f46e18456 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:20.446750 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c565147f-c12c-4a4b-bd86-b17f46e18456 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:20.447686 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c565147f-c12c-4a4b-bd86-b17f46e18456 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:20.501109 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 914/1830] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:56:20 2022] POST /identity/v3/projects => generated 383 bytes in 310 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:56:20.503497 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7272661d-7fae-4005-8454-7e1f0cf18a63 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:20.553053 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-c565147f-c12c-4a4b-bd86-b17f46e18456 None None] MFA Rules not processed for user `c9a5d45e4e6d4d38813d0884d3fbf1f1`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:20.555020 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-c565147f-c12c-4a4b-bd86-b17f46e18456 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:20.577749 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:20.586947 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=93,keystone:DELETE=4,keystone:INSERT=3 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:20.688511 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7272661d-7fae-4005-8454-7e1f0cf18a63 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:20.695322 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c17eab-bcf5-4b33-8158-949398d388ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:20.696298 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c17eab-bcf5-4b33-8158-949398d388ed admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:20.696775 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c17eab-bcf5-4b33-8158-949398d388ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:20.697238 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c17eab-bcf5-4b33-8158-949398d388ed admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:20.698141 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58c17eab-bcf5-4b33-8158-949398d388ed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:20.699883 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58c17eab-bcf5-4b33-8158-949398d388ed admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:20.746239 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 917/1831] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:20 2022] POST /identity/v3/auth/tokens => generated 3286 bytes in 308 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:20.747428 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c565147f-c12c-4a4b-bd86-b17f46e18456 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:20.761363 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 915/1832] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:20 2022] GET /identity/v3/roles => generated 1420 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:56:20.765316 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c17eab-bcf5-4b33-8158-949398d388ed admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:20.770755 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c565147f-c12c-4a4b-bd86-b17f46e18456 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:20.785416 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fbdaa0b-7558-462b-99c9-131aacc05aee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:20.786041 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbdaa0b-7558-462b-99c9-131aacc05aee admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:20.786320 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbdaa0b-7558-462b-99c9-131aacc05aee admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:20.786592 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbdaa0b-7558-462b-99c9-131aacc05aee admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:20.787231 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fbdaa0b-7558-462b-99c9-131aacc05aee admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:20.790047 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fbdaa0b-7558-462b-99c9-131aacc05aee admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:20.792477 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c17eab-bcf5-4b33-8158-949398d388ed admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:20.806510 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9680dd18-5369-414a-8a88-bddfd71b6ab1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:20.807295 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9680dd18-5369-414a-8a88-bddfd71b6ab1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:20.808047 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9680dd18-5369-414a-8a88-bddfd71b6ab1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:20.808047 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9680dd18-5369-414a-8a88-bddfd71b6ab1 admin admin] PATH_INFO: `/v3/projects/0ba384f2387f43539de482688bcccf66/users/d062f5ba56b347efab90e047e77e74e7/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:20.814868 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9680dd18-5369-414a-8a88-bddfd71b6ab1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0ba384f2387f43539de482688bcccf66, user_id=d062f5ba56b347efab90e047e77e74e7, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:20.817800 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9680dd18-5369-414a-8a88-bddfd71b6ab1 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:20.848259 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 918/1833] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:56:20 2022] POST /identity/v3/users => generated 292 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:56:20.864206 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fbdaa0b-7558-462b-99c9-131aacc05aee admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:20.865025 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 916/1834] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:56:20 2022] PUT /identity/v3/projects/0ba384f2387f43539de482688bcccf66/users/d062f5ba56b347efab90e047e77e74e7/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:20.867715 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9680dd18-5369-414a-8a88-bddfd71b6ab1 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:20.890222 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fbdaa0b-7558-462b-99c9-131aacc05aee admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:20.990917 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9680dd18-5369-414a-8a88-bddfd71b6ab1 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:20.998267 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de607683-1164-4be1-95cb-35dc284906fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:20.999296 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de607683-1164-4be1-95cb-35dc284906fb admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:20.999774 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de607683-1164-4be1-95cb-35dc284906fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:21.000114 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de607683-1164-4be1-95cb-35dc284906fb admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:21.001040 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de607683-1164-4be1-95cb-35dc284906fb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:21.004993 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de607683-1164-4be1-95cb-35dc284906fb admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:21.025874 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-340c2188-4681-4c4d-a780-052031c70bb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:21.026563 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340c2188-4681-4c4d-a780-052031c70bb0 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:21.026848 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340c2188-4681-4c4d-a780-052031c70bb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:21.027103 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340c2188-4681-4c4d-a780-052031c70bb0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:21.027741 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-340c2188-4681-4c4d-a780-052031c70bb0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:21.031914 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-340c2188-4681-4c4d-a780-052031c70bb0 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:21.033445 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 917/1835] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:20 2022] GET /identity/v3/roles => generated 1420 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:56:21.035178 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18713a6c-ab47-4459-8245-62e5bbfb0578 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:21.035703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18713a6c-ab47-4459-8245-62e5bbfb0578 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:21.036176 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18713a6c-ab47-4459-8245-62e5bbfb0578 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:21.078521 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-18713a6c-ab47-4459-8245-62e5bbfb0578 None None] MFA Rules not processed for user `d062f5ba56b347efab90e047e77e74e7`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:21.079946 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-18713a6c-ab47-4459-8245-62e5bbfb0578 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:21.097904 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 919/1836] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:56:20 2022] POST /identity/v3/projects => generated 407 bytes in 249 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:56:21.101583 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-340c2188-4681-4c4d-a780-052031c70bb0 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:21.250857 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 918/1837] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:21 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 216 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:21.252285 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18713a6c-ab47-4459-8245-62e5bbfb0578 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:21.263316 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-340c2188-4681-4c4d-a780-052031c70bb0 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:21.268317 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18713a6c-ab47-4459-8245-62e5bbfb0578 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:21.269861 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a33d68c-7f08-4e81-941d-0530bb2875ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:21.271142 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a33d68c-7f08-4e81-941d-0530bb2875ed admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:21.271142 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a33d68c-7f08-4e81-941d-0530bb2875ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:21.271344 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a33d68c-7f08-4e81-941d-0530bb2875ed admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:21.272100 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a33d68c-7f08-4e81-941d-0530bb2875ed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:21.276020 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a33d68c-7f08-4e81-941d-0530bb2875ed admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:21.278280 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c419a4f0-41c5-435a-a76e-9f9ba59fd988 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:21.279475 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c419a4f0-41c5-435a-a76e-9f9ba59fd988 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:21.279902 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c419a4f0-41c5-435a-a76e-9f9ba59fd988 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:21.280345 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c419a4f0-41c5-435a-a76e-9f9ba59fd988 admin admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:21.281140 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c419a4f0-41c5-435a-a76e-9f9ba59fd988 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:21.284387 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c419a4f0-41c5-435a-a76e-9f9ba59fd988 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:21.307065 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 920/1838] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:21 2022] GET /identity/v3/roles => generated 1420 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:56:21.307065 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a33d68c-7f08-4e81-941d-0530bb2875ed admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:21.311314 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-5a33d68c-7f08-4e81-941d-0530bb2875ed admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:21.330713 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 919/1839] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:56:21 2022] POST /identity/v3/users => generated 310 bytes in 78 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:56:21.334493 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c419a4f0-41c5-435a-a76e-9f9ba59fd988 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:21.342736 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:21.355290 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c419a4f0-41c5-435a-a76e-9f9ba59fd988 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:21.365626 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e1b33a-6baa-4568-9acc-678fbbb19a48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:21.366331 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e1b33a-6baa-4568-9acc-678fbbb19a48 admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:21.366832 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e1b33a-6baa-4568-9acc-678fbbb19a48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:21.367095 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e1b33a-6baa-4568-9acc-678fbbb19a48 admin admin] PATH_INFO: `/v3/projects/176ceec8a49a484a99c0e32ca07a29f0/users/7a3222b2c8454fedbec4a6ad886bc1b7/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:21.372746 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:21.398547 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47e1b33a-6baa-4568-9acc-678fbbb19a48 admin admin] RBAC: Authorizing `identity:create_grant(project_id=176ceec8a49a484a99c0e32ca07a29f0, user_id=7a3222b2c8454fedbec4a6ad886bc1b7, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:21.402956 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47e1b33a-6baa-4568-9acc-678fbbb19a48 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:21.422857 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 920/1840] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:56:21 2022] PUT /identity/v3/projects/176ceec8a49a484a99c0e32ca07a29f0/users/7a3222b2c8454fedbec4a6ad886bc1b7/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 92 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:21.428555 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a33d68c-7f08-4e81-941d-0530bb2875ed admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:21.434929 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e1b33a-6baa-4568-9acc-678fbbb19a48 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:21.507693 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e0c3fe-815d-459f-9daa-f678e252a937 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:21.508485 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e0c3fe-815d-459f-9daa-f678e252a937 service cinder] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:21.508983 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e0c3fe-815d-459f-9daa-f678e252a937 service cinder] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:21.509498 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e0c3fe-815d-459f-9daa-f678e252a937 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:21.532418 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e1b33a-6baa-4568-9acc-678fbbb19a48 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:21.542757 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d9a5235-a9cd-4be6-b1a0-8ffd7303e8c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:21.543510 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9a5235-a9cd-4be6-b1a0-8ffd7303e8c2 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:21.545539 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9a5235-a9cd-4be6-b1a0-8ffd7303e8c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:21.545843 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9a5235-a9cd-4be6-b1a0-8ffd7303e8c2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:21.547157 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d9a5235-a9cd-4be6-b1a0-8ffd7303e8c2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:21.548962 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d9a5235-a9cd-4be6-b1a0-8ffd7303e8c2 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:21.566765 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 921/1841] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:21 2022] GET /identity/v3/roles => generated 1420 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:56:21.566765 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a69236-907a-4124-b5b6-eacaeb1410c7 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:21.566765 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a69236-907a-4124-b5b6-eacaeb1410c7 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:21.566765 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a69236-907a-4124-b5b6-eacaeb1410c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:21.608953 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-98a69236-907a-4124-b5b6-eacaeb1410c7 None None] MFA Rules not processed for user `7a3222b2c8454fedbec4a6ad886bc1b7`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:21.611008 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-98a69236-907a-4124-b5b6-eacaeb1410c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:21.617762 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70e0c3fe-815d-459f-9daa-f678e252a937 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:21.619657 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70e0c3fe-815d-459f-9daa-f678e252a937 service cinder] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:21.634005 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 921/1842] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:56:21 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 329 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:56:21.634506 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e0c3fe-815d-459f-9daa-f678e252a937 service cinder] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:21.648205 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e0c3fe-815d-459f-9daa-f678e252a937 service cinder] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:21.655257 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b30591b-3f20-4ab7-b3bc-1343ac7496e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:21.657779 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b30591b-3f20-4ab7-b3bc-1343ac7496e1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:21.657779 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b30591b-3f20-4ab7-b3bc-1343ac7496e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:21.658001 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b30591b-3f20-4ab7-b3bc-1343ac7496e1 admin admin] PATH_INFO: `/v3/projects/b5d8eb9c319b468d97a4dec4561eb69d/users/18b184c6854d458f86301b3876d60dfb/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:21.686349 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b30591b-3f20-4ab7-b3bc-1343ac7496e1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b5d8eb9c319b468d97a4dec4561eb69d, user_id=18b184c6854d458f86301b3876d60dfb, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:21.691073 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b30591b-3f20-4ab7-b3bc-1343ac7496e1 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:21.711576 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 922/1843] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:21 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:21.712576 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a69236-907a-4124-b5b6-eacaeb1410c7 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:21.725501 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a69236-907a-4124-b5b6-eacaeb1410c7 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:21.745185 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 922/1844] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:56:21 2022] PUT /identity/v3/projects/b5d8eb9c319b468d97a4dec4561eb69d/users/18b184c6854d458f86301b3876d60dfb/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:21.749158 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b30591b-3f20-4ab7-b3bc-1343ac7496e1 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:21.818094 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-385e70d6-60df-489b-90c3-80f7d6b3058a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:21.822189 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385e70d6-60df-489b-90c3-80f7d6b3058a admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:21.822578 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385e70d6-60df-489b-90c3-80f7d6b3058a admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:21.824233 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385e70d6-60df-489b-90c3-80f7d6b3058a admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:21.826412 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-385e70d6-60df-489b-90c3-80f7d6b3058a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:21.831971 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-385e70d6-60df-489b-90c3-80f7d6b3058a admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:21.868766 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:21.881139 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:21.888523 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 923/1845] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:56:21 2022] POST /identity/v3/projects => generated 381 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:56:21.907588 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-385e70d6-60df-489b-90c3-80f7d6b3058a admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:21.925839 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b30591b-3f20-4ab7-b3bc-1343ac7496e1 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:22.074316 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-385e70d6-60df-489b-90c3-80f7d6b3058a admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:22.082338 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5490de49-ffe9-4062-9ee4-14438ea49388 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:22.084711 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5490de49-ffe9-4062-9ee4-14438ea49388 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:22.086125 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5490de49-ffe9-4062-9ee4-14438ea49388 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:22.086507 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5490de49-ffe9-4062-9ee4-14438ea49388 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:22.086678 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad80d9a-d32c-458b-9c41-aea31eb94fde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:22.087178 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5490de49-ffe9-4062-9ee4-14438ea49388 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:22.087347 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad80d9a-d32c-458b-9c41-aea31eb94fde admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:22.087610 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad80d9a-d32c-458b-9c41-aea31eb94fde admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:22.087947 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad80d9a-d32c-458b-9c41-aea31eb94fde admin admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:22.088585 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cad80d9a-d32c-458b-9c41-aea31eb94fde admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:22.092014 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cad80d9a-d32c-458b-9c41-aea31eb94fde admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:22.092258 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5490de49-ffe9-4062-9ee4-14438ea49388 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:22.134358 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 923/1846] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:21 2022] GET /identity/v3/roles => generated 1420 bytes in 385 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 00:56:22.139532 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5490de49-ffe9-4062-9ee4-14438ea49388 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:22.169943 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5490de49-ffe9-4062-9ee4-14438ea49388 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:22.188086 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e02d3c2-7d98-4f63-ad3e-051a8ead91fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:22.188086 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e02d3c2-7d98-4f63-ad3e-051a8ead91fa admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:22.188086 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e02d3c2-7d98-4f63-ad3e-051a8ead91fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:22.188086 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e02d3c2-7d98-4f63-ad3e-051a8ead91fa admin admin] PATH_INFO: `/v3/projects/b5d8eb9c319b468d97a4dec4561eb69d/users/18b184c6854d458f86301b3876d60dfb/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:22.195784 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e02d3c2-7d98-4f63-ad3e-051a8ead91fa admin admin] RBAC: Authorizing `identity:create_grant(project_id=b5d8eb9c319b468d97a4dec4561eb69d, user_id=18b184c6854d458f86301b3876d60dfb, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:22.205996 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e02d3c2-7d98-4f63-ad3e-051a8ead91fa admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:22.207116 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 924/1847] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:56:21 2022] POST /identity/v3/users => generated 297 bytes in 315 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:56:22.208055 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad80d9a-d32c-458b-9c41-aea31eb94fde admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:22.233763 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad80d9a-d32c-458b-9c41-aea31eb94fde admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:22.248350 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 924/1848] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:56:22 2022] PUT /identity/v3/projects/b5d8eb9c319b468d97a4dec4561eb69d/users/18b184c6854d458f86301b3876d60dfb/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:22.263135 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb5ff77-93c3-446f-8301-cbaeeadec5c9 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:22.263331 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb5ff77-93c3-446f-8301-cbaeeadec5c9 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:22.263560 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb5ff77-93c3-446f-8301-cbaeeadec5c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:22.347338 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-ecb5ff77-93c3-446f-8301-cbaeeadec5c9 None None] MFA Rules not processed for user `18b184c6854d458f86301b3876d60dfb`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:22.349008 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-ecb5ff77-93c3-446f-8301-cbaeeadec5c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:22.395592 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8fee18-337b-4032-8d9f-4b55b77877f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:22.396358 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8fee18-337b-4032-8d9f-4b55b77877f1 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:22.396637 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8fee18-337b-4032-8d9f-4b55b77877f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:22.396906 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8fee18-337b-4032-8d9f-4b55b77877f1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:22.397730 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b8fee18-337b-4032-8d9f-4b55b77877f1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:22.399519 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b8fee18-337b-4032-8d9f-4b55b77877f1 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:22.428558 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 925/1849] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:22 2022] GET /identity/v3/roles => generated 1420 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:56:22.430268 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8fee18-337b-4032-8d9f-4b55b77877f1 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:22.441855 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8fee18-337b-4032-8d9f-4b55b77877f1 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:22.451488 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb520d1-24b3-4b41-bc72-723af7ac923c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:22.451915 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb520d1-24b3-4b41-bc72-723af7ac923c admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:22.452191 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb520d1-24b3-4b41-bc72-723af7ac923c admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:22.452490 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb520d1-24b3-4b41-bc72-723af7ac923c admin admin] PATH_INFO: `/v3/projects/7cdbe3fc97bf4747a0f09b7fae69c462/users/12b502b3c7ec49d48e87c60d24b4abb8/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:22.511398 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eb520d1-24b3-4b41-bc72-723af7ac923c admin admin] RBAC: Authorizing `identity:create_grant(project_id=7cdbe3fc97bf4747a0f09b7fae69c462, user_id=12b502b3c7ec49d48e87c60d24b4abb8, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:22.514987 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eb520d1-24b3-4b41-bc72-723af7ac923c admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:22.536179 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 926/1850] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:56:22 2022] PUT /identity/v3/projects/7cdbe3fc97bf4747a0f09b7fae69c462/users/12b502b3c7ec49d48e87c60d24b4abb8/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:22.537692 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911e8d7a-3714-4d78-a313-8cf7123edda0 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:22.538074 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911e8d7a-3714-4d78-a313-8cf7123edda0 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:22.539022 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911e8d7a-3714-4d78-a313-8cf7123edda0 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:22.540936 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 927/1851] 10.222.0.52 () {54 vars in 921 bytes} [Sun Sep 25 00:56:22 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:56:22.542003 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-911e8d7a-3714-4d78-a313-8cf7123edda0 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:22.562191 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 925/1852] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:22 2022] POST /identity/v3/auth/tokens => generated 3351 bytes in 300 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 00:56:22.590853 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb5ff77-93c3-446f-8301-cbaeeadec5c9 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:22.592387 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-ecb5ff77-93c3-446f-8301-cbaeeadec5c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:22.631727 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:22.639044 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-911e8d7a-3714-4d78-a313-8cf7123edda0 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:22.645535 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e9de8f-cdf8-493c-ab25-4d3cf504849a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:22.646327 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e9de8f-cdf8-493c-ab25-4d3cf504849a admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:22.646851 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e9de8f-cdf8-493c-ab25-4d3cf504849a admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:22.647250 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e9de8f-cdf8-493c-ab25-4d3cf504849a admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:22.648564 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e9de8f-cdf8-493c-ab25-4d3cf504849a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:22.650458 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e9de8f-cdf8-493c-ab25-4d3cf504849a admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:22.674574 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 928/1853] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:22 2022] GET /identity/v3/roles => generated 1420 bytes in 125 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:56:22.674574 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e9de8f-cdf8-493c-ab25-4d3cf504849a admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:22.689316 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e9de8f-cdf8-493c-ab25-4d3cf504849a admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:22.709450 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04530ad5-b0ed-45ec-81bc-26a83e69f7c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:22.711424 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04530ad5-b0ed-45ec-81bc-26a83e69f7c5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:22.712153 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04530ad5-b0ed-45ec-81bc-26a83e69f7c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:22.713031 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04530ad5-b0ed-45ec-81bc-26a83e69f7c5 admin admin] PATH_INFO: `/v3/projects/7cdbe3fc97bf4747a0f09b7fae69c462/users/12b502b3c7ec49d48e87c60d24b4abb8/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:22.718179 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04530ad5-b0ed-45ec-81bc-26a83e69f7c5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7cdbe3fc97bf4747a0f09b7fae69c462, user_id=12b502b3c7ec49d48e87c60d24b4abb8, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:22.722127 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04530ad5-b0ed-45ec-81bc-26a83e69f7c5 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:22.767781 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 929/1854] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:56:22 2022] PUT /identity/v3/projects/7cdbe3fc97bf4747a0f09b7fae69c462/users/12b502b3c7ec49d48e87c60d24b4abb8/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 87 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:22.769574 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e2793e2-a333-4b72-b0f7-0be308cfe9ca None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:22.770683 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e2793e2-a333-4b72-b0f7-0be308cfe9ca None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:22.772144 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e2793e2-a333-4b72-b0f7-0be308cfe9ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:22.797699 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb5ff77-93c3-446f-8301-cbaeeadec5c9 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:22.831617 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-5e2793e2-a333-4b72-b0f7-0be308cfe9ca None None] MFA Rules not processed for user `f142988a4dd14a5f96eacea45c046f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:22.832464 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-5e2793e2-a333-4b72-b0f7-0be308cfe9ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:22.858533 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:22.927067 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20fac4a3-64b6-44d9-9637-50fb0d04999b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:22.927810 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20fac4a3-64b6-44d9-9637-50fb0d04999b service cinder] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:22.928102 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20fac4a3-64b6-44d9-9637-50fb0d04999b service cinder] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:22.928578 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20fac4a3-64b6-44d9-9637-50fb0d04999b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:22.974298 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 930/1855] 10.222.0.52 () {58 vars in 1018 bytes} [Sun Sep 25 00:56:22 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:22.975935 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e29de53-4fcb-439e-b24f-875a17463646 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:22.977054 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e29de53-4fcb-439e-b24f-875a17463646 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:22.977360 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e29de53-4fcb-439e-b24f-875a17463646 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:23.011214 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-0e29de53-4fcb-439e-b24f-875a17463646 None None] MFA Rules not processed for user `12b502b3c7ec49d48e87c60d24b4abb8`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:23.014228 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-0e29de53-4fcb-439e-b24f-875a17463646 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:23.163015 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20fac4a3-64b6-44d9-9637-50fb0d04999b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:23.164735 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20fac4a3-64b6-44d9-9637-50fb0d04999b service cinder] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:23.184230 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 926/1856] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:56:22 2022] GET /identity/v3/auth/tokens => generated 3351 bytes in 593 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:56:23.184972 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20fac4a3-64b6-44d9-9637-50fb0d04999b service cinder] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:23.189185 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-20fac4a3-64b6-44d9-9637-50fb0d04999b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:23.215615 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 931/1857] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:22 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 241 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:23.234966 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20fac4a3-64b6-44d9-9637-50fb0d04999b service cinder] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:23.237502 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e29de53-4fcb-439e-b24f-875a17463646 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:23.249285 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e29de53-4fcb-439e-b24f-875a17463646 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:23.255946 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328045b1-de5e-4a9f-a0a3-50dcd04015ee service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f142988a4dd14a5f96eacea45c046f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:23.256573 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328045b1-de5e-4a9f-a0a3-50dcd04015ee service nova] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:23.256849 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328045b1-de5e-4a9f-a0a3-50dcd04015ee service nova] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:23.258211 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5afaa1b-d4ba-4f78-9b74-1365f752a8fe service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:23.258809 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5afaa1b-d4ba-4f78-9b74-1365f752a8fe service cinder] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:23.259122 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5afaa1b-d4ba-4f78-9b74-1365f752a8fe service cinder] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:23.259392 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5afaa1b-d4ba-4f78-9b74-1365f752a8fe service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:23.260151 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328045b1-de5e-4a9f-a0a3-50dcd04015ee service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:23.287563 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5afaa1b-d4ba-4f78-9b74-1365f752a8fe service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:23.289162 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5afaa1b-d4ba-4f78-9b74-1365f752a8fe service cinder] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:23.290139 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328045b1-de5e-4a9f-a0a3-50dcd04015ee service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:23.291727 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328045b1-de5e-4a9f-a0a3-50dcd04015ee service nova] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:23.320403 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 927/1858] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:56:23 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:56:23.332537 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 932/1859] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:56:23 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 00:56:23.606788 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328045b1-de5e-4a9f-a0a3-50dcd04015ee service nova] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:23.621106 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328045b1-de5e-4a9f-a0a3-50dcd04015ee service nova] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:23.633472 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00971d04-9570-400e-ad70-31369d745e97 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:23.634268 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00971d04-9570-400e-ad70-31369d745e97 service cinder] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:23.634672 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00971d04-9570-400e-ad70-31369d745e97 service cinder] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:23.634950 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00971d04-9570-400e-ad70-31369d745e97 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:23.665758 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5afaa1b-d4ba-4f78-9b74-1365f752a8fe service cinder] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:23.678780 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5afaa1b-d4ba-4f78-9b74-1365f752a8fe service cinder] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:23.684634 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-010ad52d-793d-4cdd-929c-5b68ce92ea44 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:23.685805 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010ad52d-793d-4cdd-929c-5b68ce92ea44 service cinder] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:23.686347 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010ad52d-793d-4cdd-929c-5b68ce92ea44 service cinder] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:23.686949 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010ad52d-793d-4cdd-929c-5b68ce92ea44 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:23.745972 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00971d04-9570-400e-ad70-31369d745e97 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:23.747822 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00971d04-9570-400e-ad70-31369d745e97 service cinder] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:23.777421 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 928/1860] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:56:23 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 160 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:56:23.805498 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:23.827005 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-010ad52d-793d-4cdd-929c-5b68ce92ea44 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:23.829139 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-010ad52d-793d-4cdd-929c-5b68ce92ea44 service cinder] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:23.845221 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 933/1861] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:56:23 2022] GET /identity/v3/auth/tokens => generated 3286 bytes in 179 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:56:25.253750 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00971d04-9570-400e-ad70-31369d745e97 service cinder] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:25.360382 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:25.365609 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00971d04-9570-400e-ad70-31369d745e97 service cinder] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:25.372657 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba146c1-8c65-4ab5-9648-e9d0066d8aa0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:25.374289 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba146c1-8c65-4ab5-9648-e9d0066d8aa0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:25.374289 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba146c1-8c65-4ab5-9648-e9d0066d8aa0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:25.374289 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba146c1-8c65-4ab5-9648-e9d0066d8aa0 admin admin] PATH_INFO: `/v3/users/800fd23f307c470a849d3a92f1c3c831` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:25.375582 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba146c1-8c65-4ab5-9648-e9d0066d8aa0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=800fd23f307c470a849d3a92f1c3c831)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:25.377898 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba146c1-8c65-4ab5-9648-e9d0066d8aa0 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:25.455351 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-1ba146c1-8c65-4ab5-9648-e9d0066d8aa0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '800fd23f307c470a849d3a92f1c3c831'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:25.464305 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-1ba146c1-8c65-4ab5-9648-e9d0066d8aa0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '800fd23f307c470a849d3a92f1c3c831'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:25.465467 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-1ba146c1-8c65-4ab5-9648-e9d0066d8aa0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '800fd23f307c470a849d3a92f1c3c831'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:25.494922 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-1ba146c1-8c65-4ab5-9648-e9d0066d8aa0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '800fd23f307c470a849d3a92f1c3c831'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:25.504348 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-1ba146c1-8c65-4ab5-9648-e9d0066d8aa0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '800fd23f307c470a849d3a92f1c3c831'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:25.532030 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 929/1862] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:56:25 2022] DELETE /identity/v3/users/800fd23f307c470a849d3a92f1c3c831 => generated 0 bytes in 273 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:25.533777 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098bc7cc-42de-4727-a622-e97ba337c439 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:25.534251 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098bc7cc-42de-4727-a622-e97ba337c439 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:25.534326 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098bc7cc-42de-4727-a622-e97ba337c439 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:25.559354 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-098bc7cc-42de-4727-a622-e97ba337c439 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:25.563238 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-098bc7cc-42de-4727-a622-e97ba337c439 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:25.696243 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 934/1863] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:25.707587 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba146c1-8c65-4ab5-9648-e9d0066d8aa0 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:25.710292 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-1ba146c1-8c65-4ab5-9648-e9d0066d8aa0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:25.838389 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba146c1-8c65-4ab5-9648-e9d0066d8aa0 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:25.844548 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c062ce20-92ea-4768-b95c-fc1d14d90912 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:25.845679 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c062ce20-92ea-4768-b95c-fc1d14d90912 service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:25.846399 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c062ce20-92ea-4768-b95c-fc1d14d90912 service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:25.846728 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c062ce20-92ea-4768-b95c-fc1d14d90912 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:25.874603 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c062ce20-92ea-4768-b95c-fc1d14d90912 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:25.874603 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c062ce20-92ea-4768-b95c-fc1d14d90912 service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:25.887101 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 930/1864] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:56:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:56:26.295853 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-098bc7cc-42de-4727-a622-e97ba337c439 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:26.297203 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-098bc7cc-42de-4727-a622-e97ba337c439 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:26.312697 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-098bc7cc-42de-4727-a622-e97ba337c439 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:26.317741 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e338dd-20e5-43b9-a125-8492695417b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:26.318374 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e338dd-20e5-43b9-a125-8492695417b3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:26.318649 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e338dd-20e5-43b9-a125-8492695417b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:26.318919 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e338dd-20e5-43b9-a125-8492695417b3 admin admin] PATH_INFO: `/v3/projects/48f37c93452a40fb9f1070965497a755` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:26.320232 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e338dd-20e5-43b9-a125-8492695417b3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=48f37c93452a40fb9f1070965497a755)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:26.322192 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e338dd-20e5-43b9-a125-8492695417b3 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:26.381657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-90e338dd-20e5-43b9-a125-8492695417b3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '48f37c93452a40fb9f1070965497a755', 'request_id': 'req-90e338dd-20e5-43b9-a125-8492695417b3'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:26.390803 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-90e338dd-20e5-43b9-a125-8492695417b3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '48f37c93452a40fb9f1070965497a755', 'request_id': 'req-90e338dd-20e5-43b9-a125-8492695417b3'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:26.397664 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-90e338dd-20e5-43b9-a125-8492695417b3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '48f37c93452a40fb9f1070965497a755', 'request_id': 'req-90e338dd-20e5-43b9-a125-8492695417b3'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:26.409537 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-90e338dd-20e5-43b9-a125-8492695417b3 admin admin] The token cache is being invalidate because project 48f37c93452a40fb9f1070965497a755 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:56:26.409998 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-90e338dd-20e5-43b9-a125-8492695417b3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:26.411082 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-90e338dd-20e5-43b9-a125-8492695417b3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:26.412539 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 935/1865] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:56:26 2022] DELETE /identity/v3/projects/48f37c93452a40fb9f1070965497a755 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 00:56:26.416982 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c062ce20-92ea-4768-b95c-fc1d14d90912 service neutron] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:26.426959 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c062ce20-92ea-4768-b95c-fc1d14d90912 service neutron] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:26.505606 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0780b19f-8cdf-41a1-ba9d-8dea68b916ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:26.506243 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0780b19f-8cdf-41a1-ba9d-8dea68b916ec admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:26.506522 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0780b19f-8cdf-41a1-ba9d-8dea68b916ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:26.507325 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0780b19f-8cdf-41a1-ba9d-8dea68b916ec admin admin] PATH_INFO: `/v3/users/d062f5ba56b347efab90e047e77e74e7` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:26.508555 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0780b19f-8cdf-41a1-ba9d-8dea68b916ec admin admin] RBAC: Authorizing `identity:delete_user(user_id=d062f5ba56b347efab90e047e77e74e7)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:26.510621 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0780b19f-8cdf-41a1-ba9d-8dea68b916ec admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:26.535309 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:26.562454 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-0780b19f-8cdf-41a1-ba9d-8dea68b916ec admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd062f5ba56b347efab90e047e77e74e7'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:26.570492 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-0780b19f-8cdf-41a1-ba9d-8dea68b916ec admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd062f5ba56b347efab90e047e77e74e7'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:26.571209 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-0780b19f-8cdf-41a1-ba9d-8dea68b916ec admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd062f5ba56b347efab90e047e77e74e7'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:26.578329 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-0780b19f-8cdf-41a1-ba9d-8dea68b916ec admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd062f5ba56b347efab90e047e77e74e7'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:26.579044 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-0780b19f-8cdf-41a1-ba9d-8dea68b916ec admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd062f5ba56b347efab90e047e77e74e7'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:26.596339 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 931/1866] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:56:26 2022] DELETE /identity/v3/users/d062f5ba56b347efab90e047e77e74e7 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:26.871054 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e338dd-20e5-43b9-a125-8492695417b3 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:26.872527 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-90e338dd-20e5-43b9-a125-8492695417b3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:26.890500 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e338dd-20e5-43b9-a125-8492695417b3 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:26.952193 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be2c6c3-5205-4ad5-b2b7-ff72919a77a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:26.952743 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be2c6c3-5205-4ad5-b2b7-ff72919a77a7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:26.953021 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be2c6c3-5205-4ad5-b2b7-ff72919a77a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:26.953326 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be2c6c3-5205-4ad5-b2b7-ff72919a77a7 admin admin] PATH_INFO: `/v3/projects/0ba384f2387f43539de482688bcccf66` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:26.954436 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5be2c6c3-5205-4ad5-b2b7-ff72919a77a7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0ba384f2387f43539de482688bcccf66)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:26.956433 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5be2c6c3-5205-4ad5-b2b7-ff72919a77a7 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:26.964574 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:27.007829 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-5be2c6c3-5205-4ad5-b2b7-ff72919a77a7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0ba384f2387f43539de482688bcccf66', 'request_id': 'req-5be2c6c3-5205-4ad5-b2b7-ff72919a77a7'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:27.015519 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-5be2c6c3-5205-4ad5-b2b7-ff72919a77a7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0ba384f2387f43539de482688bcccf66', 'request_id': 'req-5be2c6c3-5205-4ad5-b2b7-ff72919a77a7'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:27.021209 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-5be2c6c3-5205-4ad5-b2b7-ff72919a77a7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0ba384f2387f43539de482688bcccf66', 'request_id': 'req-5be2c6c3-5205-4ad5-b2b7-ff72919a77a7'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:27.035032 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-5be2c6c3-5205-4ad5-b2b7-ff72919a77a7 admin admin] The token cache is being invalidate because project 0ba384f2387f43539de482688bcccf66 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:56:27.035313 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-5be2c6c3-5205-4ad5-b2b7-ff72919a77a7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:27.035988 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-5be2c6c3-5205-4ad5-b2b7-ff72919a77a7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:27.041637 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 936/1867] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:56:26 2022] DELETE /identity/v3/projects/0ba384f2387f43539de482688bcccf66 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:27.046656 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9bb924-1dd9-4513-a624-85ec10294ff9 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:27.046949 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9bb924-1dd9-4513-a624-85ec10294ff9 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:27.047800 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9bb924-1dd9-4513-a624-85ec10294ff9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:27.069348 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-ab9bb924-1dd9-4513-a624-85ec10294ff9 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:27.070741 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-ab9bb924-1dd9-4513-a624-85ec10294ff9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:27.136224 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 932/1868] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 00:56:27.150341 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ce3158-b95b-41e0-8ba8-3d41c39d2b52 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:27.150851 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ce3158-b95b-41e0-8ba8-3d41c39d2b52 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:27.151122 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ce3158-b95b-41e0-8ba8-3d41c39d2b52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:27.171465 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-23ce3158-b95b-41e0-8ba8-3d41c39d2b52 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:27.172717 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-23ce3158-b95b-41e0-8ba8-3d41c39d2b52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:27.181477 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 937/1869] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:27.185868 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab9bb924-1dd9-4513-a624-85ec10294ff9 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:27.195781 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab9bb924-1dd9-4513-a624-85ec10294ff9 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:27.200456 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd30435c-59fa-406e-861f-677839dfa1cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:27.201079 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd30435c-59fa-406e-861f-677839dfa1cc admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:27.201351 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd30435c-59fa-406e-861f-677839dfa1cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:27.202925 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd30435c-59fa-406e-861f-677839dfa1cc admin admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:27.202925 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd30435c-59fa-406e-861f-677839dfa1cc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:56:27.202925 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd30435c-59fa-406e-861f-677839dfa1cc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:27.204161 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd30435c-59fa-406e-861f-677839dfa1cc admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:27.218801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 933/1870] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 00:56:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:56:27.223158 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ce3158-b95b-41e0-8ba8-3d41c39d2b52 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:27.233744 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ce3158-b95b-41e0-8ba8-3d41c39d2b52 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:27.238940 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a4dd4c-5250-4252-9458-49761ba3bdc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:27.239546 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a4dd4c-5250-4252-9458-49761ba3bdc5 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:27.239818 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a4dd4c-5250-4252-9458-49761ba3bdc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:27.240082 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a4dd4c-5250-4252-9458-49761ba3bdc5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:27.240743 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50a4dd4c-5250-4252-9458-49761ba3bdc5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:27.243241 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50a4dd4c-5250-4252-9458-49761ba3bdc5 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:27.273644 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 938/1871] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:56:27 2022] POST /identity/v3/projects => generated 373 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 00:56:27.276163 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd30435c-59fa-406e-861f-677839dfa1cc admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:27.287275 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd30435c-59fa-406e-861f-677839dfa1cc admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:27.348296 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed6a5326-051d-4995-a6b2-8eb983191d24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:27.349109 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6a5326-051d-4995-a6b2-8eb983191d24 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:27.349764 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6a5326-051d-4995-a6b2-8eb983191d24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:27.350110 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6a5326-051d-4995-a6b2-8eb983191d24 admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:27.351162 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed6a5326-051d-4995-a6b2-8eb983191d24 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:27.353542 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed6a5326-051d-4995-a6b2-8eb983191d24 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:27.384697 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 934/1872] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:56:27 2022] POST /identity/v3/users => generated 287 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:56:27.385886 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a4dd4c-5250-4252-9458-49761ba3bdc5 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:27.398019 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a4dd4c-5250-4252-9458-49761ba3bdc5 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:27.402792 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ebaf68b-8157-45cd-a924-3bb07367ea7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:27.403385 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ebaf68b-8157-45cd-a924-3bb07367ea7b admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:27.403744 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ebaf68b-8157-45cd-a924-3bb07367ea7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:27.403914 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ebaf68b-8157-45cd-a924-3bb07367ea7b admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:27.404544 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ebaf68b-8157-45cd-a924-3bb07367ea7b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:27.406147 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ebaf68b-8157-45cd-a924-3bb07367ea7b admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:27.416233 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 939/1873] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:27 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:56:27.420260 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed6a5326-051d-4995-a6b2-8eb983191d24 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:27.430749 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed6a5326-051d-4995-a6b2-8eb983191d24 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:27.435985 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b80a6c-c37c-4c56-979e-61ce7c9df55d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:27.436595 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b80a6c-c37c-4c56-979e-61ce7c9df55d admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:27.436876 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b80a6c-c37c-4c56-979e-61ce7c9df55d admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:27.437147 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b80a6c-c37c-4c56-979e-61ce7c9df55d admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:27.437942 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35b80a6c-c37c-4c56-979e-61ce7c9df55d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:27.439536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35b80a6c-c37c-4c56-979e-61ce7c9df55d admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:27.451290 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 935/1874] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:27 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:56:27.455816 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ebaf68b-8157-45cd-a924-3bb07367ea7b admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:27.466281 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ebaf68b-8157-45cd-a924-3bb07367ea7b admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:27.470883 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a28c0af1-f072-42c4-ad44-692f3e14dcee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:27.471526 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28c0af1-f072-42c4-ad44-692f3e14dcee admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:27.471803 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28c0af1-f072-42c4-ad44-692f3e14dcee admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:27.472073 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28c0af1-f072-42c4-ad44-692f3e14dcee admin admin] PATH_INFO: `/v3/projects/89af4119fa954844a05b4b5cd47b1cd9/users/b15dfe3f6f7c4496bf7f327f6fcd2896/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:27.488359 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a28c0af1-f072-42c4-ad44-692f3e14dcee admin admin] RBAC: Authorizing `identity:create_grant(project_id=89af4119fa954844a05b4b5cd47b1cd9, user_id=b15dfe3f6f7c4496bf7f327f6fcd2896, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:27.491032 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a28c0af1-f072-42c4-ad44-692f3e14dcee admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:27.508035 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 940/1875] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:56:27 2022] PUT /identity/v3/projects/89af4119fa954844a05b4b5cd47b1cd9/users/b15dfe3f6f7c4496bf7f327f6fcd2896/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:27.518775 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb66d0a-43ae-4c4d-9816-8c1d4e92594a None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:27.519086 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb66d0a-43ae-4c4d-9816-8c1d4e92594a None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:27.519329 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb66d0a-43ae-4c4d-9816-8c1d4e92594a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:27.528194 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:DELETE=7,keystone:SELECT=89,keystone:INSERT=4 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:27.552958 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-6bb66d0a-43ae-4c4d-9816-8c1d4e92594a None None] MFA Rules not processed for user `b15dfe3f6f7c4496bf7f327f6fcd2896`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:27.554543 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-6bb66d0a-43ae-4c4d-9816-8c1d4e92594a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:27.644827 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 936/1876] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:27 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:27.655660 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a28c0af1-f072-42c4-ad44-692f3e14dcee admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:27.657797 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-a28c0af1-f072-42c4-ad44-692f3e14dcee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:27.738900 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a28c0af1-f072-42c4-ad44-692f3e14dcee admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:27.743891 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-336279e6-b535-4f57-a913-3620cd48f1d5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:27.744521 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336279e6-b535-4f57-a913-3620cd48f1d5 service cinder] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:27.747088 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336279e6-b535-4f57-a913-3620cd48f1d5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:27.747088 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336279e6-b535-4f57-a913-3620cd48f1d5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:27.760488 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-336279e6-b535-4f57-a913-3620cd48f1d5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:27.761980 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-336279e6-b535-4f57-a913-3620cd48f1d5 service cinder] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:27.771868 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 941/1877] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:56:27 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:56:27.795118 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bb66d0a-43ae-4c4d-9816-8c1d4e92594a None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:27.878160 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bb66d0a-43ae-4c4d-9816-8c1d4e92594a None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:27.885484 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c62417-af25-4c92-918b-829dc4a82d54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:27.886132 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c62417-af25-4c92-918b-829dc4a82d54 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:27.886410 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c62417-af25-4c92-918b-829dc4a82d54 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:27.886681 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c62417-af25-4c92-918b-829dc4a82d54 admin admin] PATH_INFO: `/v3/users/b15dfe3f6f7c4496bf7f327f6fcd2896` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:27.887816 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02c62417-af25-4c92-918b-829dc4a82d54 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b15dfe3f6f7c4496bf7f327f6fcd2896)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:27.889810 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02c62417-af25-4c92-918b-829dc4a82d54 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:27.941980 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-02c62417-af25-4c92-918b-829dc4a82d54 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b15dfe3f6f7c4496bf7f327f6fcd2896'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:27.951222 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-02c62417-af25-4c92-918b-829dc4a82d54 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b15dfe3f6f7c4496bf7f327f6fcd2896'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:27.951849 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-02c62417-af25-4c92-918b-829dc4a82d54 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b15dfe3f6f7c4496bf7f327f6fcd2896'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:27.963453 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-02c62417-af25-4c92-918b-829dc4a82d54 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b15dfe3f6f7c4496bf7f327f6fcd2896'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:27.964056 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-02c62417-af25-4c92-918b-829dc4a82d54 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b15dfe3f6f7c4496bf7f327f6fcd2896'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:27.984462 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 937/1878] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:56:27 2022] DELETE /identity/v3/users/b15dfe3f6f7c4496bf7f327f6fcd2896 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:27.989553 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fcaf383-8021-49d6-a5f0-823c93aac6b8 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:27.990053 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fcaf383-8021-49d6-a5f0-823c93aac6b8 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:27.990146 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fcaf383-8021-49d6-a5f0-823c93aac6b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:28.010437 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-5fcaf383-8021-49d6-a5f0-823c93aac6b8 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:28.011860 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-5fcaf383-8021-49d6-a5f0-823c93aac6b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:28.042796 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:28.086982 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 942/1879] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 00:56:28.097011 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c62417-af25-4c92-918b-829dc4a82d54 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:28.098923 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-02c62417-af25-4c92-918b-829dc4a82d54 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:28.119622 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:28.174479 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c62417-af25-4c92-918b-829dc4a82d54 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:28.179394 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-336dcd8f-d284-4961-8dbb-2393418af38c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:28.180018 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336dcd8f-d284-4961-8dbb-2393418af38c service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:28.180294 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336dcd8f-d284-4961-8dbb-2393418af38c service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:28.180555 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336dcd8f-d284-4961-8dbb-2393418af38c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:28.190807 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-336dcd8f-d284-4961-8dbb-2393418af38c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:28.192514 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-336dcd8f-d284-4961-8dbb-2393418af38c service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:28.211222 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 938/1880] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:56:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:56:28.529452 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fcaf383-8021-49d6-a5f0-823c93aac6b8 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:28.531023 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-5fcaf383-8021-49d6-a5f0-823c93aac6b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:28.555905 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fcaf383-8021-49d6-a5f0-823c93aac6b8 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:28.561534 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-059c4eaf-6a04-4f1d-b686-ff86441d2b8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:28.562182 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059c4eaf-6a04-4f1d-b686-ff86441d2b8a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:28.562463 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059c4eaf-6a04-4f1d-b686-ff86441d2b8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:28.562778 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059c4eaf-6a04-4f1d-b686-ff86441d2b8a admin admin] PATH_INFO: `/v3/projects/89af4119fa954844a05b4b5cd47b1cd9` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:28.564028 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-059c4eaf-6a04-4f1d-b686-ff86441d2b8a admin admin] RBAC: Authorizing `identity:delete_project(project_id=89af4119fa954844a05b4b5cd47b1cd9)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:28.565985 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-059c4eaf-6a04-4f1d-b686-ff86441d2b8a admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:28.614101 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-059c4eaf-6a04-4f1d-b686-ff86441d2b8a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '89af4119fa954844a05b4b5cd47b1cd9', 'request_id': 'req-059c4eaf-6a04-4f1d-b686-ff86441d2b8a'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:28.621559 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-059c4eaf-6a04-4f1d-b686-ff86441d2b8a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '89af4119fa954844a05b4b5cd47b1cd9', 'request_id': 'req-059c4eaf-6a04-4f1d-b686-ff86441d2b8a'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:28.626972 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-059c4eaf-6a04-4f1d-b686-ff86441d2b8a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '89af4119fa954844a05b4b5cd47b1cd9', 'request_id': 'req-059c4eaf-6a04-4f1d-b686-ff86441d2b8a'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:28.641991 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-059c4eaf-6a04-4f1d-b686-ff86441d2b8a admin admin] The token cache is being invalidate because project 89af4119fa954844a05b4b5cd47b1cd9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:56:28.642284 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-059c4eaf-6a04-4f1d-b686-ff86441d2b8a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:28.642934 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-059c4eaf-6a04-4f1d-b686-ff86441d2b8a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:28.644492 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 943/1881] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:56:28 2022] DELETE /identity/v3/projects/89af4119fa954844a05b4b5cd47b1cd9 => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:28.650145 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4284ee9-7085-4631-965c-ecfe5b0323d7 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:28.650702 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4284ee9-7085-4631-965c-ecfe5b0323d7 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:28.650702 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4284ee9-7085-4631-965c-ecfe5b0323d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:28.671477 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-c4284ee9-7085-4631-965c-ecfe5b0323d7 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:28.672682 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-c4284ee9-7085-4631-965c-ecfe5b0323d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:28.735740 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 939/1882] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:28.744833 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f5a147-5d4e-47b3-ab85-477f3a73d02d None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:28.745203 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f5a147-5d4e-47b3-ab85-477f3a73d02d None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:28.745647 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f5a147-5d4e-47b3-ab85-477f3a73d02d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:28.778267 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-84f5a147-5d4e-47b3-ab85-477f3a73d02d None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:28.780548 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-84f5a147-5d4e-47b3-ab85-477f3a73d02d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:28.797174 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 944/1883] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:28.800347 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4284ee9-7085-4631-965c-ecfe5b0323d7 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:28.814010 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4284ee9-7085-4631-965c-ecfe5b0323d7 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:28.819914 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c384949-ea1d-4d17-b864-81dd0e44ddee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:28.820678 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c384949-ea1d-4d17-b864-81dd0e44ddee admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:28.821044 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c384949-ea1d-4d17-b864-81dd0e44ddee admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:28.821420 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c384949-ea1d-4d17-b864-81dd0e44ddee admin admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:28.822152 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c384949-ea1d-4d17-b864-81dd0e44ddee admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:56:28.822525 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c384949-ea1d-4d17-b864-81dd0e44ddee admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:28.824211 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c384949-ea1d-4d17-b864-81dd0e44ddee admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:28.838580 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 940/1884] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 00:56:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:56:28.843183 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f5a147-5d4e-47b3-ab85-477f3a73d02d None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:28.859974 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f5a147-5d4e-47b3-ab85-477f3a73d02d None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:28.865688 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4256a8-4526-44c4-a669-1f0db67f6a4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:28.866929 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4256a8-4526-44c4-a669-1f0db67f6a4a admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:28.867353 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4256a8-4526-44c4-a669-1f0db67f6a4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:28.867994 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4256a8-4526-44c4-a669-1f0db67f6a4a admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:28.868486 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a4256a8-4526-44c4-a669-1f0db67f6a4a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:28.870623 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a4256a8-4526-44c4-a669-1f0db67f6a4a admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:28.914788 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 945/1885] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:56:28 2022] POST /identity/v3/projects => generated 377 bytes in 69 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:56:28.916211 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c384949-ea1d-4d17-b864-81dd0e44ddee admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:28.926402 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c384949-ea1d-4d17-b864-81dd0e44ddee admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:28.983824 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0934fb4c-e2a6-49cd-8617-67b2a5027183 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:28.984424 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0934fb4c-e2a6-49cd-8617-67b2a5027183 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:28.984700 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0934fb4c-e2a6-49cd-8617-67b2a5027183 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:28.984967 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0934fb4c-e2a6-49cd-8617-67b2a5027183 admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:28.985649 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0934fb4c-e2a6-49cd-8617-67b2a5027183 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:28.987617 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0934fb4c-e2a6-49cd-8617-67b2a5027183 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:29.012758 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 941/1886] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:56:28 2022] POST /identity/v3/users => generated 289 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:56:29.013869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4256a8-4526-44c4-a669-1f0db67f6a4a admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:29.024799 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4256a8-4526-44c4-a669-1f0db67f6a4a admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:29.029759 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0fd3f74-d72c-4ec3-b041-61b470cdcdeb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:29.030423 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0fd3f74-d72c-4ec3-b041-61b470cdcdeb admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:29.030758 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0fd3f74-d72c-4ec3-b041-61b470cdcdeb admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:29.031105 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0fd3f74-d72c-4ec3-b041-61b470cdcdeb admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:29.031808 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0fd3f74-d72c-4ec3-b041-61b470cdcdeb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:29.033393 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0fd3f74-d72c-4ec3-b041-61b470cdcdeb admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:29.043637 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 946/1887] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:29 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 00:56:29.048313 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0934fb4c-e2a6-49cd-8617-67b2a5027183 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:29.059125 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0934fb4c-e2a6-49cd-8617-67b2a5027183 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:29.063651 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338db27d-f772-4115-9ccd-0a3499079ed3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:29.064277 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338db27d-f772-4115-9ccd-0a3499079ed3 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:29.064551 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338db27d-f772-4115-9ccd-0a3499079ed3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:29.064810 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338db27d-f772-4115-9ccd-0a3499079ed3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:29.065478 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-338db27d-f772-4115-9ccd-0a3499079ed3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:29.067054 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-338db27d-f772-4115-9ccd-0a3499079ed3 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:29.079389 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 942/1888] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:29 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:56:29.083559 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0fd3f74-d72c-4ec3-b041-61b470cdcdeb admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:29.094156 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0fd3f74-d72c-4ec3-b041-61b470cdcdeb admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:29.098783 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81561ea8-c524-460f-a87b-96d7ddf5801f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:29.099391 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81561ea8-c524-460f-a87b-96d7ddf5801f admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:29.099662 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81561ea8-c524-460f-a87b-96d7ddf5801f admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:29.099923 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81561ea8-c524-460f-a87b-96d7ddf5801f admin admin] PATH_INFO: `/v3/projects/b61e2d35de03491fb4eea8e0c3789bf1/users/a5dfd7e712454d7db264058178c6bb7c/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:29.116513 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81561ea8-c524-460f-a87b-96d7ddf5801f admin admin] RBAC: Authorizing `identity:create_grant(project_id=b61e2d35de03491fb4eea8e0c3789bf1, user_id=a5dfd7e712454d7db264058178c6bb7c, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:29.119211 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81561ea8-c524-460f-a87b-96d7ddf5801f admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:29.134508 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 947/1889] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:56:29 2022] PUT /identity/v3/projects/b61e2d35de03491fb4eea8e0c3789bf1/users/a5dfd7e712454d7db264058178c6bb7c/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:29.145725 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05da68ce-2d05-416f-a8b1-7519e162d9cd None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:29.146167 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05da68ce-2d05-416f-a8b1-7519e162d9cd None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:29.146551 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05da68ce-2d05-416f-a8b1-7519e162d9cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:29.176424 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-05da68ce-2d05-416f-a8b1-7519e162d9cd None None] MFA Rules not processed for user `a5dfd7e712454d7db264058178c6bb7c`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:29.177699 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-05da68ce-2d05-416f-a8b1-7519e162d9cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:29.187867 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:29.246724 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 943/1890] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:29 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:29.258896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81561ea8-c524-460f-a87b-96d7ddf5801f admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:29.260551 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-81561ea8-c524-460f-a87b-96d7ddf5801f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:29.322292 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:29.343137 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81561ea8-c524-460f-a87b-96d7ddf5801f admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:29.346837 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627605f8-5f93-4c96-95c6-dc2e46c995ab service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4ebfc0abf1b842459a44ca9324252688', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:29.347432 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627605f8-5f93-4c96-95c6-dc2e46c995ab service glance] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:29.347706 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627605f8-5f93-4c96-95c6-dc2e46c995ab service glance] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:29.347973 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627605f8-5f93-4c96-95c6-dc2e46c995ab service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:29.357452 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-627605f8-5f93-4c96-95c6-dc2e46c995ab service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:29.359530 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-627605f8-5f93-4c96-95c6-dc2e46c995ab service glance] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:29.372166 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 948/1891] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:56:29 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 00:56:33.293312 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05da68ce-2d05-416f-a8b1-7519e162d9cd None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:33.294637 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-05da68ce-2d05-416f-a8b1-7519e162d9cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:33.408901 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05da68ce-2d05-416f-a8b1-7519e162d9cd None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:33.415381 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b0ec8f0-d5d2-4ac0-99cc-8f46d7b21e9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:33.416227 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0ec8f0-d5d2-4ac0-99cc-8f46d7b21e9e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:33.416787 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0ec8f0-d5d2-4ac0-99cc-8f46d7b21e9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:33.417163 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0ec8f0-d5d2-4ac0-99cc-8f46d7b21e9e admin admin] PATH_INFO: `/v3/users/7a3222b2c8454fedbec4a6ad886bc1b7` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:33.418533 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b0ec8f0-d5d2-4ac0-99cc-8f46d7b21e9e admin admin] RBAC: Authorizing `identity:delete_user(user_id=7a3222b2c8454fedbec4a6ad886bc1b7)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:33.421937 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b0ec8f0-d5d2-4ac0-99cc-8f46d7b21e9e admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:33.525299 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-7b0ec8f0-d5d2-4ac0-99cc-8f46d7b21e9e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7a3222b2c8454fedbec4a6ad886bc1b7'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:33.543737 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-7b0ec8f0-d5d2-4ac0-99cc-8f46d7b21e9e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7a3222b2c8454fedbec4a6ad886bc1b7'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:33.544621 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-7b0ec8f0-d5d2-4ac0-99cc-8f46d7b21e9e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7a3222b2c8454fedbec4a6ad886bc1b7'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:33.553686 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-7b0ec8f0-d5d2-4ac0-99cc-8f46d7b21e9e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7a3222b2c8454fedbec4a6ad886bc1b7'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:33.558074 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-7b0ec8f0-d5d2-4ac0-99cc-8f46d7b21e9e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7a3222b2c8454fedbec4a6ad886bc1b7'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:33.586712 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 944/1892] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:56:33 2022] DELETE /identity/v3/users/7a3222b2c8454fedbec4a6ad886bc1b7 => generated 0 bytes in 292 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:33.589083 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aebbe3c-7deb-4b62-b78e-e304f53aa6fa None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:33.589612 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aebbe3c-7deb-4b62-b78e-e304f53aa6fa None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:33.589693 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aebbe3c-7deb-4b62-b78e-e304f53aa6fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:33.626410 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-9aebbe3c-7deb-4b62-b78e-e304f53aa6fa None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:33.628460 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-9aebbe3c-7deb-4b62-b78e-e304f53aa6fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:33.781679 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 949/1893] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 00:56:33.794639 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b0ec8f0-d5d2-4ac0-99cc-8f46d7b21e9e admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:33.796351 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-7b0ec8f0-d5d2-4ac0-99cc-8f46d7b21e9e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:33.839471 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:33.865918 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b0ec8f0-d5d2-4ac0-99cc-8f46d7b21e9e admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:33.869833 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e32c12-a4ce-4624-970c-7f50ddeccdfa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:33.870439 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e32c12-a4ce-4624-970c-7f50ddeccdfa service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:33.870715 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e32c12-a4ce-4624-970c-7f50ddeccdfa service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:33.870979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e32c12-a4ce-4624-970c-7f50ddeccdfa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:33.882025 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e32c12-a4ce-4624-970c-7f50ddeccdfa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:33.883475 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e32c12-a4ce-4624-970c-7f50ddeccdfa service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:33.893790 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 945/1894] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:56:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:56:34.553020 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aebbe3c-7deb-4b62-b78e-e304f53aa6fa None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:34.554833 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-9aebbe3c-7deb-4b62-b78e-e304f53aa6fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:34.567189 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aebbe3c-7deb-4b62-b78e-e304f53aa6fa None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:34.572958 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab207e10-6d16-4d85-b51f-cc24ce9af1a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:34.573628 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab207e10-6d16-4d85-b51f-cc24ce9af1a0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:34.573902 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab207e10-6d16-4d85-b51f-cc24ce9af1a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:34.574171 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab207e10-6d16-4d85-b51f-cc24ce9af1a0 admin admin] PATH_INFO: `/v3/projects/176ceec8a49a484a99c0e32ca07a29f0` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:34.575305 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab207e10-6d16-4d85-b51f-cc24ce9af1a0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=176ceec8a49a484a99c0e32ca07a29f0)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:34.577949 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab207e10-6d16-4d85-b51f-cc24ce9af1a0 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:34.628293 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-ab207e10-6d16-4d85-b51f-cc24ce9af1a0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '176ceec8a49a484a99c0e32ca07a29f0', 'request_id': 'req-ab207e10-6d16-4d85-b51f-cc24ce9af1a0'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:34.636404 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-ab207e10-6d16-4d85-b51f-cc24ce9af1a0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '176ceec8a49a484a99c0e32ca07a29f0', 'request_id': 'req-ab207e10-6d16-4d85-b51f-cc24ce9af1a0'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:34.642124 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-ab207e10-6d16-4d85-b51f-cc24ce9af1a0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '176ceec8a49a484a99c0e32ca07a29f0', 'request_id': 'req-ab207e10-6d16-4d85-b51f-cc24ce9af1a0'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:34.654600 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-ab207e10-6d16-4d85-b51f-cc24ce9af1a0 admin admin] The token cache is being invalidate because project 176ceec8a49a484a99c0e32ca07a29f0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:56:34.654893 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-ab207e10-6d16-4d85-b51f-cc24ce9af1a0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:34.655557 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-ab207e10-6d16-4d85-b51f-cc24ce9af1a0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:34.657175 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 950/1895] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:56:34 2022] DELETE /identity/v3/projects/176ceec8a49a484a99c0e32ca07a29f0 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:34.661349 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e32c12-a4ce-4624-970c-7f50ddeccdfa service neutron] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:34.671355 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e32c12-a4ce-4624-970c-7f50ddeccdfa service neutron] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:34.734517 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70407853-a292-4d55-ac98-9b6789b72cb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:34.734929 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70407853-a292-4d55-ac98-9b6789b72cb9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:34.735183 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70407853-a292-4d55-ac98-9b6789b72cb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:34.735449 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70407853-a292-4d55-ac98-9b6789b72cb9 admin admin] PATH_INFO: `/v3/users/12b502b3c7ec49d48e87c60d24b4abb8` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:34.736581 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70407853-a292-4d55-ac98-9b6789b72cb9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=12b502b3c7ec49d48e87c60d24b4abb8)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:34.738814 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70407853-a292-4d55-ac98-9b6789b72cb9 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:34.811551 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-70407853-a292-4d55-ac98-9b6789b72cb9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '12b502b3c7ec49d48e87c60d24b4abb8'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:34.831065 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-70407853-a292-4d55-ac98-9b6789b72cb9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '12b502b3c7ec49d48e87c60d24b4abb8'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:34.834310 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-70407853-a292-4d55-ac98-9b6789b72cb9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '12b502b3c7ec49d48e87c60d24b4abb8'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:34.844657 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-70407853-a292-4d55-ac98-9b6789b72cb9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '12b502b3c7ec49d48e87c60d24b4abb8'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:34.845449 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-70407853-a292-4d55-ac98-9b6789b72cb9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '12b502b3c7ec49d48e87c60d24b4abb8'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:34.884773 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 946/1896] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:56:34 2022] DELETE /identity/v3/users/12b502b3c7ec49d48e87c60d24b4abb8 => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:35.156662 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab207e10-6d16-4d85-b51f-cc24ce9af1a0 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:35.162516 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-ab207e10-6d16-4d85-b51f-cc24ce9af1a0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:35.184406 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab207e10-6d16-4d85-b51f-cc24ce9af1a0 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:35.292559 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5d9bc4-9d63-4a8b-8618-8211c083598f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:35.292559 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5d9bc4-9d63-4a8b-8618-8211c083598f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:35.292559 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5d9bc4-9d63-4a8b-8618-8211c083598f admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:35.292559 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5d9bc4-9d63-4a8b-8618-8211c083598f admin admin] PATH_INFO: `/v3/users/c9a5d45e4e6d4d38813d0884d3fbf1f1` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:35.292895 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c5d9bc4-9d63-4a8b-8618-8211c083598f admin admin] RBAC: Authorizing `identity:delete_user(user_id=c9a5d45e4e6d4d38813d0884d3fbf1f1)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:35.296233 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c5d9bc4-9d63-4a8b-8618-8211c083598f admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:35.299754 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:35.379893 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-9c5d9bc4-9d63-4a8b-8618-8211c083598f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c9a5d45e4e6d4d38813d0884d3fbf1f1'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:35.381697 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-9c5d9bc4-9d63-4a8b-8618-8211c083598f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c9a5d45e4e6d4d38813d0884d3fbf1f1'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:35.401430 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-9c5d9bc4-9d63-4a8b-8618-8211c083598f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c9a5d45e4e6d4d38813d0884d3fbf1f1'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:35.402119 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-9c5d9bc4-9d63-4a8b-8618-8211c083598f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c9a5d45e4e6d4d38813d0884d3fbf1f1'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:35.410308 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-9c5d9bc4-9d63-4a8b-8618-8211c083598f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c9a5d45e4e6d4d38813d0884d3fbf1f1'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:35.436738 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 951/1897] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:56:35 2022] DELETE /identity/v3/users/c9a5d45e4e6d4d38813d0884d3fbf1f1 => generated 0 bytes in 277 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:35.438452 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74cfe275-39ec-4e0d-a599-a342f8059e91 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:35.438734 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74cfe275-39ec-4e0d-a599-a342f8059e91 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:35.439002 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74cfe275-39ec-4e0d-a599-a342f8059e91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:35.462910 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-74cfe275-39ec-4e0d-a599-a342f8059e91 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:35.464276 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-74cfe275-39ec-4e0d-a599-a342f8059e91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:35.487662 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5d9bc4-9d63-4a8b-8618-8211c083598f admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:35.490330 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-9c5d9bc4-9d63-4a8b-8618-8211c083598f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:35.554850 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:35.574363 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 947/1898] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:35.580966 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74cfe275-39ec-4e0d-a599-a342f8059e91 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:35.581558 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5d9bc4-9d63-4a8b-8618-8211c083598f admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:35.594813 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5e3f0f-ac75-4bc8-8a09-c427df99c58b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:35.595348 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5e3f0f-ac75-4bc8-8a09-c427df99c58b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:35.595601 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5e3f0f-ac75-4bc8-8a09-c427df99c58b admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:35.595978 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5e3f0f-ac75-4bc8-8a09-c427df99c58b admin admin] PATH_INFO: `/v3/projects/7cdbe3fc97bf4747a0f09b7fae69c462` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:35.597013 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5e3f0f-ac75-4bc8-8a09-c427df99c58b admin admin] RBAC: Authorizing `identity:delete_project(project_id=7cdbe3fc97bf4747a0f09b7fae69c462)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:35.597447 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-74cfe275-39ec-4e0d-a599-a342f8059e91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:35.598996 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5e3f0f-ac75-4bc8-8a09-c427df99c58b admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:35.650414 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-0d5e3f0f-ac75-4bc8-8a09-c427df99c58b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7cdbe3fc97bf4747a0f09b7fae69c462', 'request_id': 'req-0d5e3f0f-ac75-4bc8-8a09-c427df99c58b'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:35.660185 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-0d5e3f0f-ac75-4bc8-8a09-c427df99c58b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7cdbe3fc97bf4747a0f09b7fae69c462', 'request_id': 'req-0d5e3f0f-ac75-4bc8-8a09-c427df99c58b'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:35.660660 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74cfe275-39ec-4e0d-a599-a342f8059e91 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:35.668438 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-0d5e3f0f-ac75-4bc8-8a09-c427df99c58b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7cdbe3fc97bf4747a0f09b7fae69c462', 'request_id': 'req-0d5e3f0f-ac75-4bc8-8a09-c427df99c58b'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:35.681851 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-0d5e3f0f-ac75-4bc8-8a09-c427df99c58b admin admin] The token cache is being invalidate because project 7cdbe3fc97bf4747a0f09b7fae69c462 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:56:35.682402 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-0d5e3f0f-ac75-4bc8-8a09-c427df99c58b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:35.683095 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-0d5e3f0f-ac75-4bc8-8a09-c427df99c58b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:35.684772 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 952/1899] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:56:35 2022] DELETE /identity/v3/projects/7cdbe3fc97bf4747a0f09b7fae69c462 => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:35.690036 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca8678d-9631-405e-a070-cdc39ab7e8d1 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:35.690441 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca8678d-9631-405e-a070-cdc39ab7e8d1 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:35.690716 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca8678d-9631-405e-a070-cdc39ab7e8d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:35.711575 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-0ca8678d-9631-405e-a070-cdc39ab7e8d1 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:35.712769 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-0ca8678d-9631-405e-a070-cdc39ab7e8d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:35.715506 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8262312d-46a7-403c-b9e6-89f21b6edffd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:35.716143 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8262312d-46a7-403c-b9e6-89f21b6edffd service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:35.716424 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8262312d-46a7-403c-b9e6-89f21b6edffd service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:35.716804 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8262312d-46a7-403c-b9e6-89f21b6edffd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:35.718413 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:35.718562 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-8262312d-46a7-403c-b9e6-89f21b6edffd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:35.779643 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 953/1900] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 00:56:35.792712 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585e9705-2f33-41df-9b15-b81226c511b5 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:35.792712 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585e9705-2f33-41df-9b15-b81226c511b5 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:35.792712 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585e9705-2f33-41df-9b15-b81226c511b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:35.793101 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8262312d-46a7-403c-b9e6-89f21b6edffd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:35.793101 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8262312d-46a7-403c-b9e6-89f21b6edffd service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:35.802970 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 948/1901] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:56:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 222 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 00:56:35.812849 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-585e9705-2f33-41df-9b15-b81226c511b5 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:35.814096 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-585e9705-2f33-41df-9b15-b81226c511b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:35.823449 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 954/1902] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 00:56:35.827694 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8262312d-46a7-403c-b9e6-89f21b6edffd service neutron] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:35.838231 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8262312d-46a7-403c-b9e6-89f21b6edffd service neutron] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:35.843059 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7e5725b-3b93-4809-b4b9-db343487c94d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:35.843659 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e5725b-3b93-4809-b4b9-db343487c94d admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:35.843965 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e5725b-3b93-4809-b4b9-db343487c94d admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:35.844184 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e5725b-3b93-4809-b4b9-db343487c94d admin admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:35.844827 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7e5725b-3b93-4809-b4b9-db343487c94d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:56:35.845123 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7e5725b-3b93-4809-b4b9-db343487c94d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:35.846953 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7e5725b-3b93-4809-b4b9-db343487c94d admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:35.860128 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 949/1903] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 00:56:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:56:35.865385 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-585e9705-2f33-41df-9b15-b81226c511b5 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:35.875314 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-585e9705-2f33-41df-9b15-b81226c511b5 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:35.880214 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d2771f-4ff4-4ed4-9f2f-0c342b0310db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:35.880900 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d2771f-4ff4-4ed4-9f2f-0c342b0310db admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:35.881283 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d2771f-4ff4-4ed4-9f2f-0c342b0310db admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:35.882282 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d2771f-4ff4-4ed4-9f2f-0c342b0310db admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:35.883084 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1d2771f-4ff4-4ed4-9f2f-0c342b0310db admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:35.885065 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1d2771f-4ff4-4ed4-9f2f-0c342b0310db admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:35.923662 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 955/1904] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:56:35 2022] POST /identity/v3/projects => generated 367 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 00:56:35.925138 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7e5725b-3b93-4809-b4b9-db343487c94d admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:35.935744 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7e5725b-3b93-4809-b4b9-db343487c94d admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:35.997854 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:36.016834 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc085f9f-7b36-42e1-9d92-924d2c7effa2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:36.018103 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc085f9f-7b36-42e1-9d92-924d2c7effa2 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:36.018649 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc085f9f-7b36-42e1-9d92-924d2c7effa2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:36.018923 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc085f9f-7b36-42e1-9d92-924d2c7effa2 admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:36.019563 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc085f9f-7b36-42e1-9d92-924d2c7effa2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:36.021584 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc085f9f-7b36-42e1-9d92-924d2c7effa2 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:36.045362 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 950/1905] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:56:35 2022] POST /identity/v3/users => generated 284 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 00:56:36.049508 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d2771f-4ff4-4ed4-9f2f-0c342b0310db admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:36.060328 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d2771f-4ff4-4ed4-9f2f-0c342b0310db admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:36.066183 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86f505fa-d53e-404f-85cb-ccdf791dbbb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:36.067207 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f505fa-d53e-404f-85cb-ccdf791dbbb3 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:36.067676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f505fa-d53e-404f-85cb-ccdf791dbbb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:36.068243 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f505fa-d53e-404f-85cb-ccdf791dbbb3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:36.070220 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86f505fa-d53e-404f-85cb-ccdf791dbbb3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:36.072477 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86f505fa-d53e-404f-85cb-ccdf791dbbb3 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:36.085965 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 956/1906] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:36 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 00:56:36.090029 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc085f9f-7b36-42e1-9d92-924d2c7effa2 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:36.101031 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86f505fa-d53e-404f-85cb-ccdf791dbbb3 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:36.102894 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc085f9f-7b36-42e1-9d92-924d2c7effa2 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:36.104350 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-86f505fa-d53e-404f-85cb-ccdf791dbbb3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:36.108112 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db770ae-e504-41f0-b087-77394fee0f68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:36.108719 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db770ae-e504-41f0-b087-77394fee0f68 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:36.108992 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db770ae-e504-41f0-b087-77394fee0f68 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:36.109280 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db770ae-e504-41f0-b087-77394fee0f68 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:36.111693 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db770ae-e504-41f0-b087-77394fee0f68 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:36.113654 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db770ae-e504-41f0-b087-77394fee0f68 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:36.122936 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86f505fa-d53e-404f-85cb-ccdf791dbbb3 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:36.127219 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 951/1907] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:36 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 00:56:36.131320 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db770ae-e504-41f0-b087-77394fee0f68 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:36.132801 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4564d2df-92e0-43ef-919f-000ec1258de7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:36.133677 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4564d2df-92e0-43ef-919f-000ec1258de7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:36.134117 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4564d2df-92e0-43ef-919f-000ec1258de7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:36.134556 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4564d2df-92e0-43ef-919f-000ec1258de7 admin admin] PATH_INFO: `/v3/projects/f33e879b885c49fda61a093fdebbd9cd` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:36.143233 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4564d2df-92e0-43ef-919f-000ec1258de7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f33e879b885c49fda61a093fdebbd9cd)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:36.145240 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4564d2df-92e0-43ef-919f-000ec1258de7 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:36.152135 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db770ae-e504-41f0-b087-77394fee0f68 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:36.157366 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a49cf6e3-0963-43cc-9f74-bda57cbec20b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:36.158024 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49cf6e3-0963-43cc-9f74-bda57cbec20b admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:36.158362 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49cf6e3-0963-43cc-9f74-bda57cbec20b admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:36.158551 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49cf6e3-0963-43cc-9f74-bda57cbec20b admin admin] PATH_INFO: `/v3/projects/5154c6a303bf4de9a8b2a0335c442ba3/users/483bdf52e0ae4b11a6a27b0a2ed62f1b/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:36.192855 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a49cf6e3-0963-43cc-9f74-bda57cbec20b admin admin] RBAC: Authorizing `identity:create_grant(project_id=5154c6a303bf4de9a8b2a0335c442ba3, user_id=483bdf52e0ae4b11a6a27b0a2ed62f1b, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:36.198086 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a49cf6e3-0963-43cc-9f74-bda57cbec20b admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:36.216164 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-4564d2df-92e0-43ef-919f-000ec1258de7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f33e879b885c49fda61a093fdebbd9cd', 'request_id': 'req-4564d2df-92e0-43ef-919f-000ec1258de7'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:36.216742 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 952/1908] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:56:36 2022] PUT /identity/v3/projects/5154c6a303bf4de9a8b2a0335c442ba3/users/483bdf52e0ae4b11a6a27b0a2ed62f1b/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 86 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 00:56:36.226884 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5380c5c7-3b0c-4266-8acf-b4cc69f8bc93 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:36.227266 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5380c5c7-3b0c-4266-8acf-b4cc69f8bc93 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:36.227680 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5380c5c7-3b0c-4266-8acf-b4cc69f8bc93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:36.232106 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-4564d2df-92e0-43ef-919f-000ec1258de7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f33e879b885c49fda61a093fdebbd9cd', 'request_id': 'req-4564d2df-92e0-43ef-919f-000ec1258de7'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:36.239852 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-4564d2df-92e0-43ef-919f-000ec1258de7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f33e879b885c49fda61a093fdebbd9cd', 'request_id': 'req-4564d2df-92e0-43ef-919f-000ec1258de7'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:36.251985 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-4564d2df-92e0-43ef-919f-000ec1258de7 admin admin] The token cache is being invalidate because project f33e879b885c49fda61a093fdebbd9cd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:56:36.252278 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-4564d2df-92e0-43ef-919f-000ec1258de7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:36.253492 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-4564d2df-92e0-43ef-919f-000ec1258de7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:36.258863 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 957/1909] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:56:36 2022] DELETE /identity/v3/projects/f33e879b885c49fda61a093fdebbd9cd => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:36.261551 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4564d2df-92e0-43ef-919f-000ec1258de7 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:36.263189 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-4564d2df-92e0-43ef-919f-000ec1258de7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:36.275618 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-5380c5c7-3b0c-4266-8acf-b4cc69f8bc93 None None] MFA Rules not processed for user `483bdf52e0ae4b11a6a27b0a2ed62f1b`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:36.278659 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-5380c5c7-3b0c-4266-8acf-b4cc69f8bc93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:36.348851 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:36.365492 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4564d2df-92e0-43ef-919f-000ec1258de7 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:36.424652 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 953/1910] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:36 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 00:56:36.434183 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5380c5c7-3b0c-4266-8acf-b4cc69f8bc93 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:36.435592 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-5380c5c7-3b0c-4266-8acf-b4cc69f8bc93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:36.450898 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4b299d-830e-48de-a04d-cfa6f4b43ea6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:36.451840 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4b299d-830e-48de-a04d-cfa6f4b43ea6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:36.452113 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4b299d-830e-48de-a04d-cfa6f4b43ea6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:36.452381 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4b299d-830e-48de-a04d-cfa6f4b43ea6 admin admin] PATH_INFO: `/v3/users/18b184c6854d458f86301b3876d60dfb` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:36.453983 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd4b299d-830e-48de-a04d-cfa6f4b43ea6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=18b184c6854d458f86301b3876d60dfb)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:36.456029 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd4b299d-830e-48de-a04d-cfa6f4b43ea6 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:36.520107 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5380c5c7-3b0c-4266-8acf-b4cc69f8bc93 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:36.524363 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-dd4b299d-830e-48de-a04d-cfa6f4b43ea6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '18b184c6854d458f86301b3876d60dfb'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:36.525010 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-dd4b299d-830e-48de-a04d-cfa6f4b43ea6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '18b184c6854d458f86301b3876d60dfb'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:36.526021 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759d1102-4d99-462f-9866-07ecb45cc816 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:36.526712 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759d1102-4d99-462f-9866-07ecb45cc816 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:36.526995 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759d1102-4d99-462f-9866-07ecb45cc816 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:36.527265 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759d1102-4d99-462f-9866-07ecb45cc816 admin admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:36.527919 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-759d1102-4d99-462f-9866-07ecb45cc816 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:36.529977 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-759d1102-4d99-462f-9866-07ecb45cc816 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:36.535919 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-dd4b299d-830e-48de-a04d-cfa6f4b43ea6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '18b184c6854d458f86301b3876d60dfb'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:36.536938 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-dd4b299d-830e-48de-a04d-cfa6f4b43ea6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '18b184c6854d458f86301b3876d60dfb'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:36.547487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-dd4b299d-830e-48de-a04d-cfa6f4b43ea6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '18b184c6854d458f86301b3876d60dfb'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:36.565837 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 954/1911] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:56:36 2022] POST /identity/v3/projects => generated 367 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 00:56:36.570177 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759d1102-4d99-462f-9866-07ecb45cc816 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:36.572410 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-759d1102-4d99-462f-9866-07ecb45cc816 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:36.583093 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 958/1912] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:56:36 2022] DELETE /identity/v3/users/18b184c6854d458f86301b3876d60dfb => generated 0 bytes in 322 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:36.600714 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:36.672477 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759d1102-4d99-462f-9866-07ecb45cc816 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:36.746833 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a32e25-09a6-49da-b356-0e5c6fa3541e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:36.747544 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a32e25-09a6-49da-b356-0e5c6fa3541e admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:36.747701 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a32e25-09a6-49da-b356-0e5c6fa3541e admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:36.748118 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a32e25-09a6-49da-b356-0e5c6fa3541e admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:36.748762 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a32e25-09a6-49da-b356-0e5c6fa3541e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:36.751074 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a32e25-09a6-49da-b356-0e5c6fa3541e admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:36.799044 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 955/1913] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:56:36 2022] POST /identity/v3/users => generated 290 bytes in 230 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 00:56:36.802778 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4b299d-830e-48de-a04d-cfa6f4b43ea6 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:36.814367 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4b299d-830e-48de-a04d-cfa6f4b43ea6 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:36.819617 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d19503-328e-4ef2-b869-478b016c6c96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:36.820582 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d19503-328e-4ef2-b869-478b016c6c96 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:36.820867 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d19503-328e-4ef2-b869-478b016c6c96 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:36.821137 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d19503-328e-4ef2-b869-478b016c6c96 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:36.821841 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6d19503-328e-4ef2-b869-478b016c6c96 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:36.823557 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6d19503-328e-4ef2-b869-478b016c6c96 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:36.835280 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 959/1914] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:36 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 00:56:36.839484 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a32e25-09a6-49da-b356-0e5c6fa3541e admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:36.850003 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a32e25-09a6-49da-b356-0e5c6fa3541e admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:36.855588 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-279e30ed-2eec-463a-ab55-096596964fd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:36.856357 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279e30ed-2eec-463a-ab55-096596964fd8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:36.856529 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279e30ed-2eec-463a-ab55-096596964fd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:36.856798 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279e30ed-2eec-463a-ab55-096596964fd8 admin admin] PATH_INFO: `/v3/projects/94d5d7bf642f4ea38e7ae27d971b0d0c/users/8d5f00696f0d4f129b99269929a726f9/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:36.872800 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-279e30ed-2eec-463a-ab55-096596964fd8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=94d5d7bf642f4ea38e7ae27d971b0d0c, user_id=8d5f00696f0d4f129b99269929a726f9, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:36.875510 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-279e30ed-2eec-463a-ab55-096596964fd8 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:36.891304 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 956/1915] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:56:36 2022] PUT /identity/v3/projects/94d5d7bf642f4ea38e7ae27d971b0d0c/users/8d5f00696f0d4f129b99269929a726f9/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:36.895331 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d19503-328e-4ef2-b869-478b016c6c96 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:36.907793 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d19503-328e-4ef2-b869-478b016c6c96 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:36.937697 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-279e30ed-2eec-463a-ab55-096596964fd8 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:36.951626 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-279e30ed-2eec-463a-ab55-096596964fd8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:36.982680 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c2ec8e-a2c5-431f-8c9e-331c9e585af9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:36.983286 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c2ec8e-a2c5-431f-8c9e-331c9e585af9 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:36.983559 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c2ec8e-a2c5-431f-8c9e-331c9e585af9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:36.983825 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c2ec8e-a2c5-431f-8c9e-331c9e585af9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:36.984806 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89c2ec8e-a2c5-431f-8c9e-331c9e585af9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:36.986149 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89c2ec8e-a2c5-431f-8c9e-331c9e585af9 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:36.996781 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 960/1916] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:36 2022] GET /identity/v3/roles => generated 1420 bytes in 102 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:56:37.000858 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c2ec8e-a2c5-431f-8c9e-331c9e585af9 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:37.011338 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c2ec8e-a2c5-431f-8c9e-331c9e585af9 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:37.018465 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e27be9e7-07c3-4b0e-b61e-c629f8aa642f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:37.019141 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e27be9e7-07c3-4b0e-b61e-c629f8aa642f admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:37.019471 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e27be9e7-07c3-4b0e-b61e-c629f8aa642f admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:37.019750 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e27be9e7-07c3-4b0e-b61e-c629f8aa642f admin admin] PATH_INFO: `/v3/projects/94d5d7bf642f4ea38e7ae27d971b0d0c/users/8d5f00696f0d4f129b99269929a726f9/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:37.022715 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e27be9e7-07c3-4b0e-b61e-c629f8aa642f admin admin] RBAC: Authorizing `identity:create_grant(project_id=94d5d7bf642f4ea38e7ae27d971b0d0c, user_id=8d5f00696f0d4f129b99269929a726f9, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:37.025699 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e27be9e7-07c3-4b0e-b61e-c629f8aa642f admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:37.028155 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:37.038405 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 961/1917] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:56:36 2022] PUT /identity/v3/projects/94d5d7bf642f4ea38e7ae27d971b0d0c/users/8d5f00696f0d4f129b99269929a726f9/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 00:56:37.044257 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-279e30ed-2eec-463a-ab55-096596964fd8 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:37.048165 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da17a0ec-d155-4417-a0ad-ed44776437c0 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:37.048447 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da17a0ec-d155-4417-a0ad-ed44776437c0 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:37.048717 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da17a0ec-d155-4417-a0ad-ed44776437c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:37.077812 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-da17a0ec-d155-4417-a0ad-ed44776437c0 None None] MFA Rules not processed for user `8d5f00696f0d4f129b99269929a726f9`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:37.078145 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:37.079035 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-da17a0ec-d155-4417-a0ad-ed44776437c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:37.116686 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe71d39-047c-41c8-8c86-d20583c6b8ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:37.117608 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe71d39-047c-41c8-8c86-d20583c6b8ac admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:37.117886 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe71d39-047c-41c8-8c86-d20583c6b8ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:37.118153 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe71d39-047c-41c8-8c86-d20583c6b8ac admin admin] PATH_INFO: `/v3/projects/b5d8eb9c319b468d97a4dec4561eb69d` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:37.119290 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fe71d39-047c-41c8-8c86-d20583c6b8ac admin admin] RBAC: Authorizing `identity:delete_project(project_id=b5d8eb9c319b468d97a4dec4561eb69d)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:37.121188 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fe71d39-047c-41c8-8c86-d20583c6b8ac admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:37.159911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 962/1918] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:37 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 00:56:37.171866 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da17a0ec-d155-4417-a0ad-ed44776437c0 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:37.173909 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-da17a0ec-d155-4417-a0ad-ed44776437c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:37.175485 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-4fe71d39-047c-41c8-8c86-d20583c6b8ac admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b5d8eb9c319b468d97a4dec4561eb69d', 'request_id': 'req-4fe71d39-047c-41c8-8c86-d20583c6b8ac'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:37.185280 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-4fe71d39-047c-41c8-8c86-d20583c6b8ac admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b5d8eb9c319b468d97a4dec4561eb69d', 'request_id': 'req-4fe71d39-047c-41c8-8c86-d20583c6b8ac'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:37.191054 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-4fe71d39-047c-41c8-8c86-d20583c6b8ac admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b5d8eb9c319b468d97a4dec4561eb69d', 'request_id': 'req-4fe71d39-047c-41c8-8c86-d20583c6b8ac'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:37.203086 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-4fe71d39-047c-41c8-8c86-d20583c6b8ac admin admin] The token cache is being invalidate because project b5d8eb9c319b468d97a4dec4561eb69d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81386) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:56:37.203367 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-4fe71d39-047c-41c8-8c86-d20583c6b8ac admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:37.204020 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-4fe71d39-047c-41c8-8c86-d20583c6b8ac admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:37.205570 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 957/1919] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:56:36 2022] DELETE /identity/v3/projects/b5d8eb9c319b468d97a4dec4561eb69d => generated 0 bytes in 276 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 00:56:37.210268 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3da594-faa3-4f48-8324-d9fe10614ea4 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:37.210542 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3da594-faa3-4f48-8324-d9fe10614ea4 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:37.210809 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3da594-faa3-4f48-8324-d9fe10614ea4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:37.236399 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-ac3da594-faa3-4f48-8324-d9fe10614ea4 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:37.237741 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-ac3da594-faa3-4f48-8324-d9fe10614ea4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:37.243434 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da17a0ec-d155-4417-a0ad-ed44776437c0 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:37.299794 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6361d81-c3ba-4fa0-ad51-b54329dfd08e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:37.300431 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6361d81-c3ba-4fa0-ad51-b54329dfd08e service cinder] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:37.300703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6361d81-c3ba-4fa0-ad51-b54329dfd08e service cinder] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:37.300976 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6361d81-c3ba-4fa0-ad51-b54329dfd08e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:37.302801 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-e6361d81-c3ba-4fa0-ad51-b54329dfd08e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:37.308503 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 958/1920] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 00:56:37.316173 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0d0c14-bfd3-4d3b-9f8f-bffdd279e23b None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:37.316456 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0d0c14-bfd3-4d3b-9f8f-bffdd279e23b None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:37.316883 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0d0c14-bfd3-4d3b-9f8f-bffdd279e23b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:37.341030 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-1a0d0c14-bfd3-4d3b-9f8f-bffdd279e23b None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:37.343081 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-1a0d0c14-bfd3-4d3b-9f8f-bffdd279e23b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:37.353776 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 959/1921] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 00:56:37.358031 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a0d0c14-bfd3-4d3b-9f8f-bffdd279e23b None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:37.358870 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:37.372428 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a0d0c14-bfd3-4d3b-9f8f-bffdd279e23b None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:37.378397 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdcb9c00-f59c-4e23-a1b6-25e0442fb42f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:37.378933 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcb9c00-f59c-4e23-a1b6-25e0442fb42f admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:37.379206 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcb9c00-f59c-4e23-a1b6-25e0442fb42f admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:37.379480 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcb9c00-f59c-4e23-a1b6-25e0442fb42f admin admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:37.380212 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdcb9c00-f59c-4e23-a1b6-25e0442fb42f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:56:37.380432 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdcb9c00-f59c-4e23-a1b6-25e0442fb42f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:37.380899 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6361d81-c3ba-4fa0-ad51-b54329dfd08e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:37.382938 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6361d81-c3ba-4fa0-ad51-b54329dfd08e service cinder] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:37.383045 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdcb9c00-f59c-4e23-a1b6-25e0442fb42f admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:37.393656 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 963/1922] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:56:37 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 222 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 00:56:37.399243 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 960/1923] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 00:56:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:56:37.400386 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6361d81-c3ba-4fa0-ad51-b54329dfd08e service cinder] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:37.413875 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6361d81-c3ba-4fa0-ad51-b54329dfd08e service cinder] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:37.418813 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31298b4c-b42e-4557-9fec-661714dfaf98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:37.419516 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31298b4c-b42e-4557-9fec-661714dfaf98 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:37.419870 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31298b4c-b42e-4557-9fec-661714dfaf98 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:37.420207 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31298b4c-b42e-4557-9fec-661714dfaf98 admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:37.421012 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31298b4c-b42e-4557-9fec-661714dfaf98 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:37.427606 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31298b4c-b42e-4557-9fec-661714dfaf98 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:37.467220 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 964/1924] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:56:37 2022] POST /identity/v3/projects => generated 379 bytes in 67 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:56:37.472680 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdcb9c00-f59c-4e23-a1b6-25e0442fb42f admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:37.485825 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdcb9c00-f59c-4e23-a1b6-25e0442fb42f admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:37.519236 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31298b4c-b42e-4557-9fec-661714dfaf98 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:37.520988 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-31298b4c-b42e-4557-9fec-661714dfaf98 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:37.583549 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:37.612345 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31298b4c-b42e-4557-9fec-661714dfaf98 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:37.612538 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eef2c0e-3b4b-4127-a805-33d9167410b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:37.613588 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eef2c0e-3b4b-4127-a805-33d9167410b2 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:37.613918 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eef2c0e-3b4b-4127-a805-33d9167410b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:37.614239 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eef2c0e-3b4b-4127-a805-33d9167410b2 admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:37.614983 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eef2c0e-3b4b-4127-a805-33d9167410b2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:37.618958 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eef2c0e-3b4b-4127-a805-33d9167410b2 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:37.627561 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d97bf6f-d87c-4530-b67f-a16d3ceb623e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:37.628343 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d97bf6f-d87c-4530-b67f-a16d3ceb623e service cinder] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:37.628809 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d97bf6f-d87c-4530-b67f-a16d3ceb623e service cinder] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:37.629184 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d97bf6f-d87c-4530-b67f-a16d3ceb623e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:37.633515 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-6d97bf6f-d87c-4530-b67f-a16d3ceb623e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:37.652435 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 961/1925] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:56:37 2022] POST /identity/v3/users => generated 290 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:56:37.656457 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eef2c0e-3b4b-4127-a805-33d9167410b2 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:37.676520 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eef2c0e-3b4b-4127-a805-33d9167410b2 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:37.684726 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-132b79fa-2096-44a1-b371-e149a17665e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:37.685455 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-132b79fa-2096-44a1-b371-e149a17665e2 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:37.686214 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-132b79fa-2096-44a1-b371-e149a17665e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:37.686214 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-132b79fa-2096-44a1-b371-e149a17665e2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:37.686716 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-132b79fa-2096-44a1-b371-e149a17665e2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:37.688336 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-132b79fa-2096-44a1-b371-e149a17665e2 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:37.700171 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 962/1926] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:37 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:56:37.708550 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-132b79fa-2096-44a1-b371-e149a17665e2 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:37.719394 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-132b79fa-2096-44a1-b371-e149a17665e2 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:37.724498 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4590bd29-5f11-44ec-9d88-e615d840e473 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:37.725753 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4590bd29-5f11-44ec-9d88-e615d840e473 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:37.726712 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4590bd29-5f11-44ec-9d88-e615d840e473 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:37.727087 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4590bd29-5f11-44ec-9d88-e615d840e473 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:37.727846 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4590bd29-5f11-44ec-9d88-e615d840e473 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:37.729569 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4590bd29-5f11-44ec-9d88-e615d840e473 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:37.732430 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d97bf6f-d87c-4530-b67f-a16d3ceb623e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:37.734121 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d97bf6f-d87c-4530-b67f-a16d3ceb623e service cinder] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:37.740884 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 963/1927] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:37 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 00:56:37.747081 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4590bd29-5f11-44ec-9d88-e615d840e473 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:37.753668 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 965/1928] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:56:37 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 235 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:56:37.758908 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4590bd29-5f11-44ec-9d88-e615d840e473 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:37.764240 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6dc2c2e-e60f-4df0-ba20-c74816754841 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:37.765028 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6dc2c2e-e60f-4df0-ba20-c74816754841 admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:37.765434 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6dc2c2e-e60f-4df0-ba20-c74816754841 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:37.765767 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6dc2c2e-e60f-4df0-ba20-c74816754841 admin admin] PATH_INFO: `/v3/projects/7c9056114d8848b190a75c2de6727864/users/c7c4fcbaa6cb4e7dbd1429c4e384c7d1/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:37.783832 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6dc2c2e-e60f-4df0-ba20-c74816754841 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7c9056114d8848b190a75c2de6727864, user_id=c7c4fcbaa6cb4e7dbd1429c4e384c7d1, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:37.788847 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6dc2c2e-e60f-4df0-ba20-c74816754841 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:37.814817 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 964/1929] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:56:37 2022] PUT /identity/v3/projects/7c9056114d8848b190a75c2de6727864/users/c7c4fcbaa6cb4e7dbd1429c4e384c7d1/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:37.825528 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98493e9d-1a42-40cd-aad0-8f088331c3b9 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:37.826008 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98493e9d-1a42-40cd-aad0-8f088331c3b9 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:37.826008 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98493e9d-1a42-40cd-aad0-8f088331c3b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:37.864930 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-98493e9d-1a42-40cd-aad0-8f088331c3b9 None None] MFA Rules not processed for user `c7c4fcbaa6cb4e7dbd1429c4e384c7d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:37.866714 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-98493e9d-1a42-40cd-aad0-8f088331c3b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:37.948605 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 966/1930] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:37 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:37.953177 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6dc2c2e-e60f-4df0-ba20-c74816754841 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:38.031538 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6dc2c2e-e60f-4df0-ba20-c74816754841 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:38.036583 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1944f9fa-295f-4a8d-8b32-06a5130f98d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:38.037318 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1944f9fa-295f-4a8d-8b32-06a5130f98d7 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:38.037713 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1944f9fa-295f-4a8d-8b32-06a5130f98d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:38.038208 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1944f9fa-295f-4a8d-8b32-06a5130f98d7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:38.039024 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1944f9fa-295f-4a8d-8b32-06a5130f98d7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:38.041030 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1944f9fa-295f-4a8d-8b32-06a5130f98d7 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:38.077124 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 965/1931] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:56:37 2022] POST /identity/v3/projects => generated 379 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:56:38.080910 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98493e9d-1a42-40cd-aad0-8f088331c3b9 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:38.097012 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98493e9d-1a42-40cd-aad0-8f088331c3b9 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:38.103248 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:38.172214 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c902f3ec-01a6-40c1-aafd-20b8da3e897a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:38.172873 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c902f3ec-01a6-40c1-aafd-20b8da3e897a admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:38.173191 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c902f3ec-01a6-40c1-aafd-20b8da3e897a admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:38.173501 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c902f3ec-01a6-40c1-aafd-20b8da3e897a admin admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:38.174143 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c902f3ec-01a6-40c1-aafd-20b8da3e897a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:38.176123 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c902f3ec-01a6-40c1-aafd-20b8da3e897a admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:38.199006 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 967/1932] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:56:38 2022] POST /identity/v3/users => generated 290 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:56:38.203227 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1944f9fa-295f-4a8d-8b32-06a5130f98d7 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:38.214156 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1944f9fa-295f-4a8d-8b32-06a5130f98d7 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:38.219422 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b43564-1111-4578-94aa-858fd58e6275 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:38.220017 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b43564-1111-4578-94aa-858fd58e6275 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:38.220294 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b43564-1111-4578-94aa-858fd58e6275 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:38.220557 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b43564-1111-4578-94aa-858fd58e6275 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:38.221196 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b43564-1111-4578-94aa-858fd58e6275 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:38.222806 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b43564-1111-4578-94aa-858fd58e6275 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:38.233974 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 966/1933] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:38 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:56:38.238292 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c902f3ec-01a6-40c1-aafd-20b8da3e897a admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:38.249024 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c902f3ec-01a6-40c1-aafd-20b8da3e897a admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:38.253678 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b951bad5-3225-4836-9d0e-06e9b8657e17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:38.254259 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b951bad5-3225-4836-9d0e-06e9b8657e17 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:38.254563 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b951bad5-3225-4836-9d0e-06e9b8657e17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:38.254835 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b951bad5-3225-4836-9d0e-06e9b8657e17 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:38.255475 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b951bad5-3225-4836-9d0e-06e9b8657e17 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:38.257284 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b951bad5-3225-4836-9d0e-06e9b8657e17 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:38.267507 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 968/1934] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:38 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:56:38.271832 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b43564-1111-4578-94aa-858fd58e6275 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:38.282096 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b43564-1111-4578-94aa-858fd58e6275 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:38.287020 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cffd4100-f418-44c7-9851-5d0035c3e4db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:38.288313 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cffd4100-f418-44c7-9851-5d0035c3e4db admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:38.288598 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cffd4100-f418-44c7-9851-5d0035c3e4db admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:38.288867 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cffd4100-f418-44c7-9851-5d0035c3e4db admin admin] PATH_INFO: `/v3/projects/d1d652f52ade4b488b76564ecb719bd5/users/8e54ba9498c64f35bd7c4e47a24f5e60/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:38.304247 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cffd4100-f418-44c7-9851-5d0035c3e4db admin admin] RBAC: Authorizing `identity:create_grant(project_id=d1d652f52ade4b488b76564ecb719bd5, user_id=8e54ba9498c64f35bd7c4e47a24f5e60, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:38.306918 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cffd4100-f418-44c7-9851-5d0035c3e4db admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:38.319680 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 967/1935] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:56:38 2022] PUT /identity/v3/projects/d1d652f52ade4b488b76564ecb719bd5/users/8e54ba9498c64f35bd7c4e47a24f5e60/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:38.330837 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9241a5bd-711a-4562-8326-95b8ac7ceff4 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:38.331124 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9241a5bd-711a-4562-8326-95b8ac7ceff4 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:38.331397 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9241a5bd-711a-4562-8326-95b8ac7ceff4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:38.360807 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-9241a5bd-711a-4562-8326-95b8ac7ceff4 None None] MFA Rules not processed for user `8e54ba9498c64f35bd7c4e47a24f5e60`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:38.362198 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-9241a5bd-711a-4562-8326-95b8ac7ceff4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:38.432285 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 969/1936] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:38 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:38.436800 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cffd4100-f418-44c7-9851-5d0035c3e4db admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:38.479311 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:38.501175 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cffd4100-f418-44c7-9851-5d0035c3e4db admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:38.505854 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1989a508-16ce-44d7-9bf6-5e84c9073dd2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:38.507008 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1989a508-16ce-44d7-9bf6-5e84c9073dd2 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:38.507393 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1989a508-16ce-44d7-9bf6-5e84c9073dd2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:38.507666 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1989a508-16ce-44d7-9bf6-5e84c9073dd2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:38.508301 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1989a508-16ce-44d7-9bf6-5e84c9073dd2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:38.510238 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1989a508-16ce-44d7-9bf6-5e84c9073dd2 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:38.536793 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 968/1937] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:56:38 2022] POST /identity/v3/projects => generated 379 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 00:56:38.541179 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9241a5bd-711a-4562-8326-95b8ac7ceff4 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:38.551879 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9241a5bd-711a-4562-8326-95b8ac7ceff4 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:38.612321 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-005d942f-6317-4e9c-b8ea-31691611b3eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:38.613134 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005d942f-6317-4e9c-b8ea-31691611b3eb admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:38.613450 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005d942f-6317-4e9c-b8ea-31691611b3eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:38.613732 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005d942f-6317-4e9c-b8ea-31691611b3eb admin admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:38.614364 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-005d942f-6317-4e9c-b8ea-31691611b3eb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:38.616924 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-005d942f-6317-4e9c-b8ea-31691611b3eb admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:38.639964 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 970/1938] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:56:38 2022] POST /identity/v3/users => generated 296 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:56:38.643830 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1989a508-16ce-44d7-9bf6-5e84c9073dd2 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:38.655645 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1989a508-16ce-44d7-9bf6-5e84c9073dd2 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:38.660544 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52934774-0689-4046-a262-f99a250194e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:38.661159 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52934774-0689-4046-a262-f99a250194e6 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:38.661489 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52934774-0689-4046-a262-f99a250194e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:38.661760 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52934774-0689-4046-a262-f99a250194e6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:38.662400 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52934774-0689-4046-a262-f99a250194e6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:38.664731 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52934774-0689-4046-a262-f99a250194e6 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:38.680473 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 969/1939] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:38 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:56:38.681803 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-005d942f-6317-4e9c-b8ea-31691611b3eb admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:38.689789 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:38.697652 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-005d942f-6317-4e9c-b8ea-31691611b3eb admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:38.706430 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38fa836e-215d-43c7-9dd9-bf2bb2710be9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:38.707269 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38fa836e-215d-43c7-9dd9-bf2bb2710be9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:38.707752 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38fa836e-215d-43c7-9dd9-bf2bb2710be9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:38.708211 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38fa836e-215d-43c7-9dd9-bf2bb2710be9 admin admin] PATH_INFO: `/v3/projects/ef8dd72bc2e0480aac0f1b8ad50ec274/users/f1f779bdd5f04ceaaf6aad226ad2a3fa/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:38.727670 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38fa836e-215d-43c7-9dd9-bf2bb2710be9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ef8dd72bc2e0480aac0f1b8ad50ec274, user_id=f1f779bdd5f04ceaaf6aad226ad2a3fa, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:38.730622 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38fa836e-215d-43c7-9dd9-bf2bb2710be9 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:38.750630 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 971/1940] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:56:38 2022] PUT /identity/v3/projects/ef8dd72bc2e0480aac0f1b8ad50ec274/users/f1f779bdd5f04ceaaf6aad226ad2a3fa/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:38.753620 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52934774-0689-4046-a262-f99a250194e6 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:38.771178 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52934774-0689-4046-a262-f99a250194e6 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:38.877649 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd951327-1a77-487f-a5e1-5e70f205a847 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:38.878443 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd951327-1a77-487f-a5e1-5e70f205a847 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:38.878825 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd951327-1a77-487f-a5e1-5e70f205a847 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:38.879219 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd951327-1a77-487f-a5e1-5e70f205a847 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:38.880012 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd951327-1a77-487f-a5e1-5e70f205a847 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:38.883176 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd951327-1a77-487f-a5e1-5e70f205a847 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:38.902486 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 970/1941] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:38 2022] GET /identity/v3/roles => generated 1420 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:56:38.904164 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38fa836e-215d-43c7-9dd9-bf2bb2710be9 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:38.924060 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38fa836e-215d-43c7-9dd9-bf2bb2710be9 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:38.933858 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae6d804-b5e4-4a7e-b630-5ca9496048ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:38.935614 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae6d804-b5e4-4a7e-b630-5ca9496048ed admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:38.936073 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae6d804-b5e4-4a7e-b630-5ca9496048ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:38.936480 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae6d804-b5e4-4a7e-b630-5ca9496048ed admin admin] PATH_INFO: `/v3/projects/ef8dd72bc2e0480aac0f1b8ad50ec274/users/f1f779bdd5f04ceaaf6aad226ad2a3fa/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:38.940876 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ae6d804-b5e4-4a7e-b630-5ca9496048ed admin admin] RBAC: Authorizing `identity:create_grant(project_id=ef8dd72bc2e0480aac0f1b8ad50ec274, user_id=f1f779bdd5f04ceaaf6aad226ad2a3fa, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:38.947276 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ae6d804-b5e4-4a7e-b630-5ca9496048ed admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:38.967153 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 972/1942] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:56:38 2022] PUT /identity/v3/projects/ef8dd72bc2e0480aac0f1b8ad50ec274/users/f1f779bdd5f04ceaaf6aad226ad2a3fa/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:38.978218 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26cf7309-c873-48bf-ad52-bbb501702cb8 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:38.979316 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26cf7309-c873-48bf-ad52-bbb501702cb8 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:38.979684 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26cf7309-c873-48bf-ad52-bbb501702cb8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:39.021412 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-26cf7309-c873-48bf-ad52-bbb501702cb8 None None] MFA Rules not processed for user `f1f779bdd5f04ceaaf6aad226ad2a3fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:39.023781 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-26cf7309-c873-48bf-ad52-bbb501702cb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:39.127057 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 971/1943] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:38 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:39.161439 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae6d804-b5e4-4a7e-b630-5ca9496048ed admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:39.313715 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae6d804-b5e4-4a7e-b630-5ca9496048ed admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:39.321966 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-658d76ac-40d5-4eee-8bcb-182d15993803 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:39.322598 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658d76ac-40d5-4eee-8bcb-182d15993803 service cinder] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:39.323329 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658d76ac-40d5-4eee-8bcb-182d15993803 service cinder] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:39.323329 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658d76ac-40d5-4eee-8bcb-182d15993803 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:39.354754 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-658d76ac-40d5-4eee-8bcb-182d15993803 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:39.356211 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-658d76ac-40d5-4eee-8bcb-182d15993803 service cinder] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:39.388023 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 973/1944] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:56:39 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 227 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:56:39.521702 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26cf7309-c873-48bf-ad52-bbb501702cb8 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:39.545440 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26cf7309-c873-48bf-ad52-bbb501702cb8 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:39.554003 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd63478-25aa-49bb-933a-84c5f6b0bdaf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:39.555191 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd63478-25aa-49bb-933a-84c5f6b0bdaf service cinder] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:39.555712 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd63478-25aa-49bb-933a-84c5f6b0bdaf service cinder] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:39.556089 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd63478-25aa-49bb-933a-84c5f6b0bdaf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:39.609377 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:39.649816 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcd63478-25aa-49bb-933a-84c5f6b0bdaf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:39.651404 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcd63478-25aa-49bb-933a-84c5f6b0bdaf service cinder] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:39.664331 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 972/1945] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:56:39 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:56:41.901598 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49ddcb9-887a-4f37-9a68-ef54c93a594e None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:41.903161 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49ddcb9-887a-4f37-9a68-ef54c93a594e None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:41.903487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49ddcb9-887a-4f37-9a68-ef54c93a594e None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:41.904955 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 974/1946] 10.222.0.52 () {54 vars in 921 bytes} [Sun Sep 25 00:56:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:56:41.916345 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac3596c-5422-458f-8e75-9c8920bde532 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:41.916812 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac3596c-5422-458f-8e75-9c8920bde532 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:41.916983 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac3596c-5422-458f-8e75-9c8920bde532 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:41.940602 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-0ac3596c-5422-458f-8e75-9c8920bde532 None None] MFA Rules not processed for user `f142988a4dd14a5f96eacea45c046f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:41.941235 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-0ac3596c-5422-458f-8e75-9c8920bde532 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:42.027677 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 973/1947] 10.222.0.52 () {58 vars in 1018 bytes} [Sun Sep 25 00:56:41 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:42.112265 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d49ddcb9-887a-4f37-9a68-ef54c93a594e None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:42.115975 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-d49ddcb9-887a-4f37-9a68-ef54c93a594e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:42.145185 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d49ddcb9-887a-4f37-9a68-ef54c93a594e None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:42.152238 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e429e3-e0ec-4538-81d6-40ebc088f5e4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f142988a4dd14a5f96eacea45c046f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:42.153235 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e429e3-e0ec-4538-81d6-40ebc088f5e4 service nova] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:42.153683 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e429e3-e0ec-4538-81d6-40ebc088f5e4 service nova] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:42.154142 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e429e3-e0ec-4538-81d6-40ebc088f5e4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:42.165630 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27e429e3-e0ec-4538-81d6-40ebc088f5e4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:42.167516 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27e429e3-e0ec-4538-81d6-40ebc088f5e4 service nova] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:42.183594 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 975/1948] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:56:42 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 72 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:56:45.639815 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac3596c-5422-458f-8e75-9c8920bde532 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:45.641312 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-0ac3596c-5422-458f-8e75-9c8920bde532 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:45.723658 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac3596c-5422-458f-8e75-9c8920bde532 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:45.729685 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a25537-c91b-42f5-be79-d03f06e929bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:45.730324 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a25537-c91b-42f5-be79-d03f06e929bc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:45.730600 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a25537-c91b-42f5-be79-d03f06e929bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:45.730866 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a25537-c91b-42f5-be79-d03f06e929bc admin admin] PATH_INFO: `/v3/users/483bdf52e0ae4b11a6a27b0a2ed62f1b` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:45.732137 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a25537-c91b-42f5-be79-d03f06e929bc admin admin] RBAC: Authorizing `identity:delete_user(user_id=483bdf52e0ae4b11a6a27b0a2ed62f1b)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:45.734408 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a25537-c91b-42f5-be79-d03f06e929bc admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:45.805467 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-54a25537-c91b-42f5-be79-d03f06e929bc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '483bdf52e0ae4b11a6a27b0a2ed62f1b'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:45.814612 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-54a25537-c91b-42f5-be79-d03f06e929bc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '483bdf52e0ae4b11a6a27b0a2ed62f1b'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:45.815394 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-54a25537-c91b-42f5-be79-d03f06e929bc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '483bdf52e0ae4b11a6a27b0a2ed62f1b'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:45.824157 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-54a25537-c91b-42f5-be79-d03f06e929bc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '483bdf52e0ae4b11a6a27b0a2ed62f1b'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:45.825463 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-54a25537-c91b-42f5-be79-d03f06e929bc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '483bdf52e0ae4b11a6a27b0a2ed62f1b'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:45.831289 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:45.850860 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 974/1949] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:56:45 2022] DELETE /identity/v3/users/483bdf52e0ae4b11a6a27b0a2ed62f1b => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:45.855454 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ee9654-45d7-40ab-b598-3b3575975317 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:45.856033 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ee9654-45d7-40ab-b598-3b3575975317 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:45.856033 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ee9654-45d7-40ab-b598-3b3575975317 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:45.882969 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-67ee9654-45d7-40ab-b598-3b3575975317 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:45.884215 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-67ee9654-45d7-40ab-b598-3b3575975317 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:45.972309 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 976/1950] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 00:56:45.985647 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a25537-c91b-42f5-be79-d03f06e929bc admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:45.987779 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-54a25537-c91b-42f5-be79-d03f06e929bc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:46.049913 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ee9654-45d7-40ab-b598-3b3575975317 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:46.051303 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-67ee9654-45d7-40ab-b598-3b3575975317 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:46.058702 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a25537-c91b-42f5-be79-d03f06e929bc admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:46.062522 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e1bf63-0ee5-49a9-8109-75c4c994385e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:46.063819 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e1bf63-0ee5-49a9-8109-75c4c994385e service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:46.064348 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e1bf63-0ee5-49a9-8109-75c4c994385e service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:46.064599 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e1bf63-0ee5-49a9-8109-75c4c994385e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:46.067094 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ee9654-45d7-40ab-b598-3b3575975317 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:46.072621 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94e7a914-81a4-4f7a-9e2e-dc3fd4cb3e59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:46.073424 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e7a914-81a4-4f7a-9e2e-dc3fd4cb3e59 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:46.073770 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e7a914-81a4-4f7a-9e2e-dc3fd4cb3e59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:46.074161 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e7a914-81a4-4f7a-9e2e-dc3fd4cb3e59 admin admin] PATH_INFO: `/v3/users/c7c4fcbaa6cb4e7dbd1429c4e384c7d1` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:46.075689 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94e7a914-81a4-4f7a-9e2e-dc3fd4cb3e59 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c7c4fcbaa6cb4e7dbd1429c4e384c7d1)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:46.076795 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2e1bf63-0ee5-49a9-8109-75c4c994385e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:46.077828 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94e7a914-81a4-4f7a-9e2e-dc3fd4cb3e59 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:46.078323 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2e1bf63-0ee5-49a9-8109-75c4c994385e service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:46.089778 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 975/1951] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:56:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:56:46.198453 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-94e7a914-81a4-4f7a-9e2e-dc3fd4cb3e59 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c7c4fcbaa6cb4e7dbd1429c4e384c7d1'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:46.199157 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-94e7a914-81a4-4f7a-9e2e-dc3fd4cb3e59 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c7c4fcbaa6cb4e7dbd1429c4e384c7d1'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:46.204402 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=89,keystone:INSERT=3,keystone:DELETE=8 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:46.210502 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-94e7a914-81a4-4f7a-9e2e-dc3fd4cb3e59 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c7c4fcbaa6cb4e7dbd1429c4e384c7d1'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:46.211439 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-94e7a914-81a4-4f7a-9e2e-dc3fd4cb3e59 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c7c4fcbaa6cb4e7dbd1429c4e384c7d1'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:46.225684 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-94e7a914-81a4-4f7a-9e2e-dc3fd4cb3e59 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c7c4fcbaa6cb4e7dbd1429c4e384c7d1'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:46.265595 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 977/1952] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:56:46 2022] DELETE /identity/v3/users/c7c4fcbaa6cb4e7dbd1429c4e384c7d1 => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:46.270210 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727bf049-e331-4725-82ae-e03c9b0fe9fe None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:46.270566 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727bf049-e331-4725-82ae-e03c9b0fe9fe None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:46.271462 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727bf049-e331-4725-82ae-e03c9b0fe9fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:46.314854 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-727bf049-e331-4725-82ae-e03c9b0fe9fe None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:46.316161 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-727bf049-e331-4725-82ae-e03c9b0fe9fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:46.430996 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 976/1953] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:46.436828 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94e7a914-81a4-4f7a-9e2e-dc3fd4cb3e59 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:46.438315 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-94e7a914-81a4-4f7a-9e2e-dc3fd4cb3e59 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:46.527256 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94e7a914-81a4-4f7a-9e2e-dc3fd4cb3e59 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:46.532137 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727bf049-e331-4725-82ae-e03c9b0fe9fe None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:46.533563 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-727bf049-e331-4725-82ae-e03c9b0fe9fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:46.539081 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-232408f1-208f-479f-81ba-75ba703745e1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:46.540603 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232408f1-208f-479f-81ba-75ba703745e1 service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:46.540940 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232408f1-208f-479f-81ba-75ba703745e1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:46.541247 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232408f1-208f-479f-81ba-75ba703745e1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:46.551170 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727bf049-e331-4725-82ae-e03c9b0fe9fe None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:46.553431 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-232408f1-208f-479f-81ba-75ba703745e1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:46.555538 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-232408f1-208f-479f-81ba-75ba703745e1 service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:46.566253 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d126bfda-6652-4d81-9e73-9ee1daa18b9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:46.569538 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d126bfda-6652-4d81-9e73-9ee1daa18b9e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:46.569833 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d126bfda-6652-4d81-9e73-9ee1daa18b9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:46.570116 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d126bfda-6652-4d81-9e73-9ee1daa18b9e admin admin] PATH_INFO: `/v3/projects/5154c6a303bf4de9a8b2a0335c442ba3` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:46.573041 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 978/1954] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:56:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:56:46.573175 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d126bfda-6652-4d81-9e73-9ee1daa18b9e admin admin] RBAC: Authorizing `identity:delete_project(project_id=5154c6a303bf4de9a8b2a0335c442ba3)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:46.575573 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d126bfda-6652-4d81-9e73-9ee1daa18b9e admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:46.653739 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-d126bfda-6652-4d81-9e73-9ee1daa18b9e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5154c6a303bf4de9a8b2a0335c442ba3', 'request_id': 'req-d126bfda-6652-4d81-9e73-9ee1daa18b9e'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:46.680340 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-d126bfda-6652-4d81-9e73-9ee1daa18b9e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5154c6a303bf4de9a8b2a0335c442ba3', 'request_id': 'req-d126bfda-6652-4d81-9e73-9ee1daa18b9e'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:46.710658 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-d126bfda-6652-4d81-9e73-9ee1daa18b9e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5154c6a303bf4de9a8b2a0335c442ba3', 'request_id': 'req-d126bfda-6652-4d81-9e73-9ee1daa18b9e'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:46.743701 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-d126bfda-6652-4d81-9e73-9ee1daa18b9e admin admin] The token cache is being invalidate because project 5154c6a303bf4de9a8b2a0335c442ba3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81386) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:56:46.757072 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-d126bfda-6652-4d81-9e73-9ee1daa18b9e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:46.759804 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-d126bfda-6652-4d81-9e73-9ee1daa18b9e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:46.764280 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 977/1955] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:56:46 2022] DELETE /identity/v3/projects/5154c6a303bf4de9a8b2a0335c442ba3 => generated 0 bytes in 233 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 00:56:46.768909 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-232408f1-208f-479f-81ba-75ba703745e1 service neutron] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:46.798945 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-232408f1-208f-479f-81ba-75ba703745e1 service neutron] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:46.982767 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce37e67-ff41-44b2-9979-0268fe01c9f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:46.984332 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce37e67-ff41-44b2-9979-0268fe01c9f1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:46.985092 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce37e67-ff41-44b2-9979-0268fe01c9f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:46.985708 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce37e67-ff41-44b2-9979-0268fe01c9f1 admin admin] PATH_INFO: `/v3/users/8d5f00696f0d4f129b99269929a726f9` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:46.988104 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce37e67-ff41-44b2-9979-0268fe01c9f1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8d5f00696f0d4f129b99269929a726f9)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:46.991747 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce37e67-ff41-44b2-9979-0268fe01c9f1 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:47.126111 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-3ce37e67-ff41-44b2-9979-0268fe01c9f1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8d5f00696f0d4f129b99269929a726f9'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:47.127261 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-3ce37e67-ff41-44b2-9979-0268fe01c9f1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8d5f00696f0d4f129b99269929a726f9'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:47.143677 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-3ce37e67-ff41-44b2-9979-0268fe01c9f1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8d5f00696f0d4f129b99269929a726f9'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:47.144967 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-3ce37e67-ff41-44b2-9979-0268fe01c9f1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8d5f00696f0d4f129b99269929a726f9'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:47.163385 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-3ce37e67-ff41-44b2-9979-0268fe01c9f1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8d5f00696f0d4f129b99269929a726f9'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:47.233794 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:47.251212 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d126bfda-6652-4d81-9e73-9ee1daa18b9e admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:47.253723 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-d126bfda-6652-4d81-9e73-9ee1daa18b9e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:47.268499 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 979/1956] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:56:46 2022] DELETE /identity/v3/users/8d5f00696f0d4f129b99269929a726f9 => generated 0 bytes in 500 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:47.316603 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d126bfda-6652-4d81-9e73-9ee1daa18b9e admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:47.345710 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:DELETE=9,keystone:SELECT=90,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:47.493241 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cacfd52-bebb-4b42-a9b0-fca1964c3586 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:47.494008 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cacfd52-bebb-4b42-a9b0-fca1964c3586 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:47.494292 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cacfd52-bebb-4b42-a9b0-fca1964c3586 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:47.497473 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cacfd52-bebb-4b42-a9b0-fca1964c3586 admin admin] PATH_INFO: `/v3/projects/7c9056114d8848b190a75c2de6727864` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:47.512558 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cacfd52-bebb-4b42-a9b0-fca1964c3586 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7c9056114d8848b190a75c2de6727864)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:47.519331 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cacfd52-bebb-4b42-a9b0-fca1964c3586 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:47.650073 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-6cacfd52-bebb-4b42-a9b0-fca1964c3586 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7c9056114d8848b190a75c2de6727864', 'request_id': 'req-6cacfd52-bebb-4b42-a9b0-fca1964c3586'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:47.659801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-6cacfd52-bebb-4b42-a9b0-fca1964c3586 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7c9056114d8848b190a75c2de6727864', 'request_id': 'req-6cacfd52-bebb-4b42-a9b0-fca1964c3586'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:47.668598 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-6cacfd52-bebb-4b42-a9b0-fca1964c3586 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7c9056114d8848b190a75c2de6727864', 'request_id': 'req-6cacfd52-bebb-4b42-a9b0-fca1964c3586'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:47.696506 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-6cacfd52-bebb-4b42-a9b0-fca1964c3586 admin admin] The token cache is being invalidate because project 7c9056114d8848b190a75c2de6727864 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81386) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:56:47.697113 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-6cacfd52-bebb-4b42-a9b0-fca1964c3586 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:47.698009 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-6cacfd52-bebb-4b42-a9b0-fca1964c3586 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:47.700635 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 978/1957] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:56:47 2022] DELETE /identity/v3/projects/7c9056114d8848b190a75c2de6727864 => generated 0 bytes in 451 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 00:56:47.704882 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce37e67-ff41-44b2-9979-0268fe01c9f1 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:47.706383 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-3ce37e67-ff41-44b2-9979-0268fe01c9f1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:47.726447 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce37e67-ff41-44b2-9979-0268fe01c9f1 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:47.762619 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cacfd52-bebb-4b42-a9b0-fca1964c3586 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:47.764551 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-6cacfd52-bebb-4b42-a9b0-fca1964c3586 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:47.869165 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a082f99f-a39b-443c-b715-8cc9931708fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:47.869845 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a082f99f-a39b-443c-b715-8cc9931708fb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:47.870127 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a082f99f-a39b-443c-b715-8cc9931708fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:47.870720 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a082f99f-a39b-443c-b715-8cc9931708fb admin admin] PATH_INFO: `/v3/users/8e54ba9498c64f35bd7c4e47a24f5e60` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:47.872761 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a082f99f-a39b-443c-b715-8cc9931708fb admin admin] RBAC: Authorizing `identity:delete_user(user_id=8e54ba9498c64f35bd7c4e47a24f5e60)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:47.874739 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a082f99f-a39b-443c-b715-8cc9931708fb admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:47.897061 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cacfd52-bebb-4b42-a9b0-fca1964c3586 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:47.902975 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4626613b-af57-4267-99be-42cf0b080805 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:47.903876 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4626613b-af57-4267-99be-42cf0b080805 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:47.903987 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4626613b-af57-4267-99be-42cf0b080805 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:47.904344 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4626613b-af57-4267-99be-42cf0b080805 admin admin] PATH_INFO: `/v3/projects/94d5d7bf642f4ea38e7ae27d971b0d0c` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:47.906822 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4626613b-af57-4267-99be-42cf0b080805 admin admin] RBAC: Authorizing `identity:delete_project(project_id=94d5d7bf642f4ea38e7ae27d971b0d0c)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:47.909405 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4626613b-af57-4267-99be-42cf0b080805 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:47.984060 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-a082f99f-a39b-443c-b715-8cc9931708fb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8e54ba9498c64f35bd7c4e47a24f5e60'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:47.985523 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-a082f99f-a39b-443c-b715-8cc9931708fb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8e54ba9498c64f35bd7c4e47a24f5e60'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:47.995421 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:47.997391 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-4626613b-af57-4267-99be-42cf0b080805 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '94d5d7bf642f4ea38e7ae27d971b0d0c', 'request_id': 'req-4626613b-af57-4267-99be-42cf0b080805'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:47.999504 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-a082f99f-a39b-443c-b715-8cc9931708fb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8e54ba9498c64f35bd7c4e47a24f5e60'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:48.000176 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-a082f99f-a39b-443c-b715-8cc9931708fb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8e54ba9498c64f35bd7c4e47a24f5e60'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:48.012119 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-a082f99f-a39b-443c-b715-8cc9931708fb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8e54ba9498c64f35bd7c4e47a24f5e60'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:48.012351 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-4626613b-af57-4267-99be-42cf0b080805 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '94d5d7bf642f4ea38e7ae27d971b0d0c', 'request_id': 'req-4626613b-af57-4267-99be-42cf0b080805'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:48.018892 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-4626613b-af57-4267-99be-42cf0b080805 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '94d5d7bf642f4ea38e7ae27d971b0d0c', 'request_id': 'req-4626613b-af57-4267-99be-42cf0b080805'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:48.030650 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-4626613b-af57-4267-99be-42cf0b080805 admin admin] The token cache is being invalidate because project 94d5d7bf642f4ea38e7ae27d971b0d0c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81386) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:56:48.030952 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-4626613b-af57-4267-99be-42cf0b080805 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:48.031625 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-4626613b-af57-4267-99be-42cf0b080805 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:48.032073 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 980/1958] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:56:47 2022] DELETE /identity/v3/users/8e54ba9498c64f35bd7c4e47a24f5e60 => generated 0 bytes in 328 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:48.033630 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 979/1959] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:56:47 2022] DELETE /identity/v3/projects/94d5d7bf642f4ea38e7ae27d971b0d0c => generated 0 bytes in 272 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 00:56:48.039348 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be7c471-381a-4bd4-a84e-64d58f6956dd None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:48.039634 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be7c471-381a-4bd4-a84e-64d58f6956dd None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:48.039905 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be7c471-381a-4bd4-a84e-64d58f6956dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:48.060123 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-7be7c471-381a-4bd4-a84e-64d58f6956dd None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:48.061369 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-7be7c471-381a-4bd4-a84e-64d58f6956dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:48.141648 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 981/1960] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 00:56:48.144563 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e2e6b6-ae61-401e-89ff-aa70a3e20797 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:48.144563 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e2e6b6-ae61-401e-89ff-aa70a3e20797 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:48.144810 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e2e6b6-ae61-401e-89ff-aa70a3e20797 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:48.168822 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-87e2e6b6-ae61-401e-89ff-aa70a3e20797 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:48.170589 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-87e2e6b6-ae61-401e-89ff-aa70a3e20797 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:48.183155 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 980/1961] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:48.184700 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be7c471-381a-4bd4-a84e-64d58f6956dd None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:48.200428 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be7c471-381a-4bd4-a84e-64d58f6956dd None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:48.222443 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e9b51e-6ea2-4f38-b2ca-46e465e71545 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:48.224327 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e9b51e-6ea2-4f38-b2ca-46e465e71545 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:48.224707 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e9b51e-6ea2-4f38-b2ca-46e465e71545 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:48.224989 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e9b51e-6ea2-4f38-b2ca-46e465e71545 admin admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:48.227861 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e9b51e-6ea2-4f38-b2ca-46e465e71545 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:56:48.228178 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e9b51e-6ea2-4f38-b2ca-46e465e71545 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:48.234910 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e9b51e-6ea2-4f38-b2ca-46e465e71545 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:48.245514 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:48.254933 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 982/1962] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 00:56:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:56:48.260379 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e2e6b6-ae61-401e-89ff-aa70a3e20797 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:48.272347 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e2e6b6-ae61-401e-89ff-aa70a3e20797 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:48.284356 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ddc703-27de-4bac-9328-b34af3b056db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:48.285086 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ddc703-27de-4bac-9328-b34af3b056db admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:48.285469 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ddc703-27de-4bac-9328-b34af3b056db admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:48.285807 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ddc703-27de-4bac-9328-b34af3b056db admin admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:48.286594 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85ddc703-27de-4bac-9328-b34af3b056db admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:48.289077 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85ddc703-27de-4bac-9328-b34af3b056db admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:48.342258 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 981/1963] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:56:48 2022] POST /identity/v3/projects => generated 383 bytes in 80 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 00:56:48.343492 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e9b51e-6ea2-4f38-b2ca-46e465e71545 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:48.359181 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e9b51e-6ea2-4f38-b2ca-46e465e71545 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:48.456965 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f257552c-f1c6-4208-a7de-b01386695c7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:48.456965 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f257552c-f1c6-4208-a7de-b01386695c7e admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:48.456965 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f257552c-f1c6-4208-a7de-b01386695c7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:48.456965 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f257552c-f1c6-4208-a7de-b01386695c7e admin admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:48.457565 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f257552c-f1c6-4208-a7de-b01386695c7e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:48.460169 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f257552c-f1c6-4208-a7de-b01386695c7e admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:48.481325 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ddc703-27de-4bac-9328-b34af3b056db admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:48.487384 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-85ddc703-27de-4bac-9328-b34af3b056db admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:48.502159 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ddc703-27de-4bac-9328-b34af3b056db admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:48.507312 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 983/1964] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:56:48 2022] POST /identity/v3/users => generated 292 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:56:48.508446 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f9b0332-a541-42f2-b639-4bfd44c3e32a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:48.508916 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f257552c-f1c6-4208-a7de-b01386695c7e admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:48.509303 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9b0332-a541-42f2-b639-4bfd44c3e32a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:48.509585 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9b0332-a541-42f2-b639-4bfd44c3e32a admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:48.509942 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9b0332-a541-42f2-b639-4bfd44c3e32a admin admin] PATH_INFO: `/v3/projects/d1d652f52ade4b488b76564ecb719bd5` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:48.511172 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f9b0332-a541-42f2-b639-4bfd44c3e32a admin admin] RBAC: Authorizing `identity:delete_project(project_id=d1d652f52ade4b488b76564ecb719bd5)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:48.513164 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f9b0332-a541-42f2-b639-4bfd44c3e32a admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:48.519965 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f257552c-f1c6-4208-a7de-b01386695c7e admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:48.525699 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a972f1ee-3c3f-4d01-8f09-215ce5f5a557 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:48.526313 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a972f1ee-3c3f-4d01-8f09-215ce5f5a557 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:48.526593 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a972f1ee-3c3f-4d01-8f09-215ce5f5a557 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:48.526882 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a972f1ee-3c3f-4d01-8f09-215ce5f5a557 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:48.527582 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a972f1ee-3c3f-4d01-8f09-215ce5f5a557 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:48.529175 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a972f1ee-3c3f-4d01-8f09-215ce5f5a557 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:48.541478 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 984/1965] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:48 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 00:56:48.545391 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a972f1ee-3c3f-4d01-8f09-215ce5f5a557 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:48.556503 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a972f1ee-3c3f-4d01-8f09-215ce5f5a557 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:48.576132 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-1f9b0332-a541-42f2-b639-4bfd44c3e32a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd1d652f52ade4b488b76564ecb719bd5', 'request_id': 'req-1f9b0332-a541-42f2-b639-4bfd44c3e32a'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:48.585543 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-1f9b0332-a541-42f2-b639-4bfd44c3e32a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd1d652f52ade4b488b76564ecb719bd5', 'request_id': 'req-1f9b0332-a541-42f2-b639-4bfd44c3e32a'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:48.592778 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-1f9b0332-a541-42f2-b639-4bfd44c3e32a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd1d652f52ade4b488b76564ecb719bd5', 'request_id': 'req-1f9b0332-a541-42f2-b639-4bfd44c3e32a'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:48.607135 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-1f9b0332-a541-42f2-b639-4bfd44c3e32a admin admin] The token cache is being invalidate because project d1d652f52ade4b488b76564ecb719bd5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81386) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:56:48.607995 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-1f9b0332-a541-42f2-b639-4bfd44c3e32a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:48.608973 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-1f9b0332-a541-42f2-b639-4bfd44c3e32a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:48.612002 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 982/1966] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:56:48 2022] DELETE /identity/v3/projects/d1d652f52ade4b488b76564ecb719bd5 => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:48.616994 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f9b0332-a541-42f2-b639-4bfd44c3e32a admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:48.618495 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-1f9b0332-a541-42f2-b639-4bfd44c3e32a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:48.630473 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3ef5bed-413c-49d0-8c32-dbce4f5532ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:48.631660 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ef5bed-413c-49d0-8c32-dbce4f5532ce admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:48.632116 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ef5bed-413c-49d0-8c32-dbce4f5532ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:48.632389 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ef5bed-413c-49d0-8c32-dbce4f5532ce admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:48.633141 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3ef5bed-413c-49d0-8c32-dbce4f5532ce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:48.634782 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3ef5bed-413c-49d0-8c32-dbce4f5532ce admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:48.645058 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 985/1967] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:48 2022] GET /identity/v3/roles => generated 1420 bytes in 100 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 00:56:48.649102 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3ef5bed-413c-49d0-8c32-dbce4f5532ce admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:48.651032 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-d3ef5bed-413c-49d0-8c32-dbce4f5532ce admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:48.664480 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3ef5bed-413c-49d0-8c32-dbce4f5532ce admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:48.669729 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c98afc6c-7cf0-4d78-829a-e65f549df2af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:48.670497 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c98afc6c-7cf0-4d78-829a-e65f549df2af admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:48.670931 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c98afc6c-7cf0-4d78-829a-e65f549df2af admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:48.671267 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c98afc6c-7cf0-4d78-829a-e65f549df2af admin admin] PATH_INFO: `/v3/projects/949e5c4b9d274f488df36e640d3b9eec/users/c1cf6ec8b164449c8799da78a5f0a909/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:48.688916 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c98afc6c-7cf0-4d78-829a-e65f549df2af admin admin] RBAC: Authorizing `identity:create_grant(project_id=949e5c4b9d274f488df36e640d3b9eec, user_id=c1cf6ec8b164449c8799da78a5f0a909, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:48.692022 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c98afc6c-7cf0-4d78-829a-e65f549df2af admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:48.698712 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f9b0332-a541-42f2-b639-4bfd44c3e32a admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:48.704241 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfdbec5d-8aa5-4901-b06b-030fa48070b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:48.704964 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfdbec5d-8aa5-4901-b06b-030fa48070b0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:48.705364 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfdbec5d-8aa5-4901-b06b-030fa48070b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:48.705698 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfdbec5d-8aa5-4901-b06b-030fa48070b0 admin admin] PATH_INFO: `/v3/users/f1f779bdd5f04ceaaf6aad226ad2a3fa` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:48.706938 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfdbec5d-8aa5-4901-b06b-030fa48070b0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f1f779bdd5f04ceaaf6aad226ad2a3fa)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:48.709057 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfdbec5d-8aa5-4901-b06b-030fa48070b0 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:48.712313 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 986/1968] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:56:48 2022] PUT /identity/v3/projects/949e5c4b9d274f488df36e640d3b9eec/users/c1cf6ec8b164449c8799da78a5f0a909/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:48.720540 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e8cd56d-4d8b-4017-ac66-f2ef8f555285 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:48.720999 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e8cd56d-4d8b-4017-ac66-f2ef8f555285 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:48.721371 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e8cd56d-4d8b-4017-ac66-f2ef8f555285 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:48.746214 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=81,keystone:DELETE=16,keystone:INSERT=3 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:48.756874 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-9e8cd56d-4d8b-4017-ac66-f2ef8f555285 None None] MFA Rules not processed for user `c1cf6ec8b164449c8799da78a5f0a909`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:48.758138 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-9e8cd56d-4d8b-4017-ac66-f2ef8f555285 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:48.773527 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-cfdbec5d-8aa5-4901-b06b-030fa48070b0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f1f779bdd5f04ceaaf6aad226ad2a3fa'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:48.782436 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-cfdbec5d-8aa5-4901-b06b-030fa48070b0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f1f779bdd5f04ceaaf6aad226ad2a3fa'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:48.783066 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-cfdbec5d-8aa5-4901-b06b-030fa48070b0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f1f779bdd5f04ceaaf6aad226ad2a3fa'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:48.792330 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-cfdbec5d-8aa5-4901-b06b-030fa48070b0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f1f779bdd5f04ceaaf6aad226ad2a3fa'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:48.793558 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-cfdbec5d-8aa5-4901-b06b-030fa48070b0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f1f779bdd5f04ceaaf6aad226ad2a3fa'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:48.808753 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:48.820522 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 983/1969] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:56:48 2022] DELETE /identity/v3/users/f1f779bdd5f04ceaaf6aad226ad2a3fa => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 00:56:48.854379 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 987/1970] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:48 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 00:56:48.858417 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfdbec5d-8aa5-4901-b06b-030fa48070b0 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:48.859960 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-cfdbec5d-8aa5-4901-b06b-030fa48070b0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:48.940989 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfdbec5d-8aa5-4901-b06b-030fa48070b0 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:48.947965 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aae259f-dbdd-4f1c-a78d-94e67f67aea3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:48.948814 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aae259f-dbdd-4f1c-a78d-94e67f67aea3 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:48.949100 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aae259f-dbdd-4f1c-a78d-94e67f67aea3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:48.949402 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aae259f-dbdd-4f1c-a78d-94e67f67aea3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:48.950154 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aae259f-dbdd-4f1c-a78d-94e67f67aea3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:48.953035 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aae259f-dbdd-4f1c-a78d-94e67f67aea3 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:49.010248 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 984/1971] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:56:48 2022] POST /identity/v3/projects => generated 383 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:56:49.011242 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e8cd56d-4d8b-4017-ac66-f2ef8f555285 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:49.127239 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e8cd56d-4d8b-4017-ac66-f2ef8f555285 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:49.217743 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b979658-6ff4-4f90-a8c8-7a71d5f89846 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:49.218468 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b979658-6ff4-4f90-a8c8-7a71d5f89846 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:49.218819 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b979658-6ff4-4f90-a8c8-7a71d5f89846 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:49.220006 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b979658-6ff4-4f90-a8c8-7a71d5f89846 admin admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:49.220439 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b979658-6ff4-4f90-a8c8-7a71d5f89846 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:49.222649 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b979658-6ff4-4f90-a8c8-7a71d5f89846 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:49.264824 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 988/1972] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:56:49 2022] POST /identity/v3/users => generated 298 bytes in 254 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 00:56:49.269671 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aae259f-dbdd-4f1c-a78d-94e67f67aea3 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:49.282176 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aae259f-dbdd-4f1c-a78d-94e67f67aea3 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:49.287946 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82203966-bcf1-4988-b119-4dbfa32753be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:49.288460 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82203966-bcf1-4988-b119-4dbfa32753be admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:49.288765 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82203966-bcf1-4988-b119-4dbfa32753be admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:49.289006 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82203966-bcf1-4988-b119-4dbfa32753be admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:49.289693 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82203966-bcf1-4988-b119-4dbfa32753be admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:49.291573 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82203966-bcf1-4988-b119-4dbfa32753be admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:49.304642 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 985/1973] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:49 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:56:49.309998 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b979658-6ff4-4f90-a8c8-7a71d5f89846 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:49.322825 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b979658-6ff4-4f90-a8c8-7a71d5f89846 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:49.327915 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82203966-bcf1-4988-b119-4dbfa32753be admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:49.329749 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-82203966-bcf1-4988-b119-4dbfa32753be admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:49.331144 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b25c89-c4ef-4266-8154-dbece3c2f7f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:49.332021 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b25c89-c4ef-4266-8154-dbece3c2f7f9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:49.332340 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b25c89-c4ef-4266-8154-dbece3c2f7f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:49.332672 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b25c89-c4ef-4266-8154-dbece3c2f7f9 admin admin] PATH_INFO: `/v3/projects/56d0ea5794f14f8d96f6b48d16fa108e/users/088623bec3a64727add0c5f1729241d7/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:49.356528 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82203966-bcf1-4988-b119-4dbfa32753be admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:49.373563 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0b25c89-c4ef-4266-8154-dbece3c2f7f9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=56d0ea5794f14f8d96f6b48d16fa108e, user_id=088623bec3a64727add0c5f1729241d7, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:49.374075 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69534b4b-0272-4208-8f7f-d28cfe320865 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:49.374075 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69534b4b-0272-4208-8f7f-d28cfe320865 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:49.374075 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69534b4b-0272-4208-8f7f-d28cfe320865 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:49.374075 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69534b4b-0272-4208-8f7f-d28cfe320865 admin admin] PATH_INFO: `/v3/projects/ef8dd72bc2e0480aac0f1b8ad50ec274` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:49.374834 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69534b4b-0272-4208-8f7f-d28cfe320865 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ef8dd72bc2e0480aac0f1b8ad50ec274)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:49.375176 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0b25c89-c4ef-4266-8154-dbece3c2f7f9 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:49.378389 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69534b4b-0272-4208-8f7f-d28cfe320865 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:49.405680 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 989/1974] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:56:49 2022] PUT /identity/v3/projects/56d0ea5794f14f8d96f6b48d16fa108e/users/088623bec3a64727add0c5f1729241d7/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 92 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:49.406718 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b25c89-c4ef-4266-8154-dbece3c2f7f9 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:49.449476 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-69534b4b-0272-4208-8f7f-d28cfe320865 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ef8dd72bc2e0480aac0f1b8ad50ec274', 'request_id': 'req-69534b4b-0272-4208-8f7f-d28cfe320865'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:49.462696 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-69534b4b-0272-4208-8f7f-d28cfe320865 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ef8dd72bc2e0480aac0f1b8ad50ec274', 'request_id': 'req-69534b4b-0272-4208-8f7f-d28cfe320865'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:49.490365 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:49.494060 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-69534b4b-0272-4208-8f7f-d28cfe320865 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ef8dd72bc2e0480aac0f1b8ad50ec274', 'request_id': 'req-69534b4b-0272-4208-8f7f-d28cfe320865'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:49.524363 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b25c89-c4ef-4266-8154-dbece3c2f7f9 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:49.536917 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-69534b4b-0272-4208-8f7f-d28cfe320865 admin admin] The token cache is being invalidate because project ef8dd72bc2e0480aac0f1b8ad50ec274 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81386) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:56:49.538121 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-69534b4b-0272-4208-8f7f-d28cfe320865 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:49.538980 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-69534b4b-0272-4208-8f7f-d28cfe320865 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:49.547468 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 986/1975] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:56:49 2022] DELETE /identity/v3/projects/ef8dd72bc2e0480aac0f1b8ad50ec274 => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:49.548951 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08037d6-55b6-4d00-8d1d-aea31a91ae76 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:49.549224 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08037d6-55b6-4d00-8d1d-aea31a91ae76 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:49.550936 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08037d6-55b6-4d00-8d1d-aea31a91ae76 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:49.574859 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=85,keystone:DELETE=12,keystone:INSERT=3 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:49.593205 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-c08037d6-55b6-4d00-8d1d-aea31a91ae76 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:49.594705 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-c08037d6-55b6-4d00-8d1d-aea31a91ae76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:49.644549 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc2a91ad-8856-4738-ad5a-2ab8099baf88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:49.647078 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2a91ad-8856-4738-ad5a-2ab8099baf88 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:49.647549 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2a91ad-8856-4738-ad5a-2ab8099baf88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:49.647798 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2a91ad-8856-4738-ad5a-2ab8099baf88 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:49.648706 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc2a91ad-8856-4738-ad5a-2ab8099baf88 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:49.651066 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc2a91ad-8856-4738-ad5a-2ab8099baf88 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:49.672942 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 990/1976] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:49 2022] GET /identity/v3/roles => generated 1420 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:56:49.674131 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc2a91ad-8856-4738-ad5a-2ab8099baf88 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:49.677608 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-bc2a91ad-8856-4738-ad5a-2ab8099baf88 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:49.689968 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 987/1977] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:49.702228 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a322aa42-c4a3-49f9-aa21-6a2ab79e4d71 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:49.702838 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a322aa42-c4a3-49f9-aa21-6a2ab79e4d71 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:49.702838 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a322aa42-c4a3-49f9-aa21-6a2ab79e4d71 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:49.707790 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc2a91ad-8856-4738-ad5a-2ab8099baf88 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:49.713419 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f121d342-ee91-4268-88d0-671990d09452 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:49.714098 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f121d342-ee91-4268-88d0-671990d09452 admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:49.714376 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f121d342-ee91-4268-88d0-671990d09452 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:49.714648 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f121d342-ee91-4268-88d0-671990d09452 admin admin] PATH_INFO: `/v3/projects/56d0ea5794f14f8d96f6b48d16fa108e/users/088623bec3a64727add0c5f1729241d7/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:49.716852 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f121d342-ee91-4268-88d0-671990d09452 admin admin] RBAC: Authorizing `identity:create_grant(project_id=56d0ea5794f14f8d96f6b48d16fa108e, user_id=088623bec3a64727add0c5f1729241d7, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:49.722583 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f121d342-ee91-4268-88d0-671990d09452 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:49.736666 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-a322aa42-c4a3-49f9-aa21-6a2ab79e4d71 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:49.742143 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-a322aa42-c4a3-49f9-aa21-6a2ab79e4d71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:49.762514 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 991/1978] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:56:49 2022] PUT /identity/v3/projects/56d0ea5794f14f8d96f6b48d16fa108e/users/088623bec3a64727add0c5f1729241d7/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 88 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:49.775036 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98291c1-323f-4cd6-b705-111fb0bf63e5 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:49.775403 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98291c1-323f-4cd6-b705-111fb0bf63e5 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:49.776028 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98291c1-323f-4cd6-b705-111fb0bf63e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:49.818351 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-b98291c1-323f-4cd6-b705-111fb0bf63e5 None None] MFA Rules not processed for user `088623bec3a64727add0c5f1729241d7`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:49.820294 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-b98291c1-323f-4cd6-b705-111fb0bf63e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:49.859718 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 988/1979] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:49.861616 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a322aa42-c4a3-49f9-aa21-6a2ab79e4d71 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:49.873309 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a322aa42-c4a3-49f9-aa21-6a2ab79e4d71 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:49.879384 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a88b9b-6128-4b28-804b-9364dcd7ce66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:49.880004 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a88b9b-6128-4b28-804b-9364dcd7ce66 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:49.880279 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a88b9b-6128-4b28-804b-9364dcd7ce66 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:49.880548 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a88b9b-6128-4b28-804b-9364dcd7ce66 admin admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:49.881460 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77a88b9b-6128-4b28-804b-9364dcd7ce66 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:56:49.881935 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77a88b9b-6128-4b28-804b-9364dcd7ce66 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:49.885177 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77a88b9b-6128-4b28-804b-9364dcd7ce66 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:49.923592 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 989/1980] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 00:56:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:56:49.924989 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a88b9b-6128-4b28-804b-9364dcd7ce66 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:49.939477 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a88b9b-6128-4b28-804b-9364dcd7ce66 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:49.946621 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bcddb6d-df10-4261-9431-b13fc108cadc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:49.947376 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bcddb6d-df10-4261-9431-b13fc108cadc admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:49.947768 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bcddb6d-df10-4261-9431-b13fc108cadc admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:49.948269 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bcddb6d-df10-4261-9431-b13fc108cadc admin admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:49.949030 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bcddb6d-df10-4261-9431-b13fc108cadc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:49.952621 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bcddb6d-df10-4261-9431-b13fc108cadc admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:49.960203 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 992/1981] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:49 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:49.968793 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b98291c1-323f-4cd6-b705-111fb0bf63e5 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:49.970394 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-b98291c1-323f-4cd6-b705-111fb0bf63e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:50.015542 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 990/1982] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:56:49 2022] POST /identity/v3/projects => generated 373 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 00:56:50.019722 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bcddb6d-df10-4261-9431-b13fc108cadc admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:50.075248 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b98291c1-323f-4cd6-b705-111fb0bf63e5 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:50.110984 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:50.129999 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bcddb6d-df10-4261-9431-b13fc108cadc admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:50.137787 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1033a123-85dd-4c37-9c07-6a72ee778522 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:50.138759 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1033a123-85dd-4c37-9c07-6a72ee778522 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:50.139033 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1033a123-85dd-4c37-9c07-6a72ee778522 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:50.139298 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1033a123-85dd-4c37-9c07-6a72ee778522 admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:50.139939 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1033a123-85dd-4c37-9c07-6a72ee778522 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:50.143530 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1033a123-85dd-4c37-9c07-6a72ee778522 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:50.160043 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdebbda8-0ffb-49e4-ad35-00e4fcbef947 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:50.160807 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdebbda8-0ffb-49e4-ad35-00e4fcbef947 service cinder] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:50.161120 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdebbda8-0ffb-49e4-ad35-00e4fcbef947 service cinder] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:50.161557 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdebbda8-0ffb-49e4-ad35-00e4fcbef947 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:50.165386 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:50.172918 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 991/1983] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:56:50 2022] POST /identity/v3/users => generated 287 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 00:56:50.174399 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1033a123-85dd-4c37-9c07-6a72ee778522 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:50.178488 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-1033a123-85dd-4c37-9c07-6a72ee778522 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:50.193007 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1033a123-85dd-4c37-9c07-6a72ee778522 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:50.199449 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1536c070-8e49-452f-be1b-8e0b3061323d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:50.200086 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1536c070-8e49-452f-be1b-8e0b3061323d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:50.200364 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1536c070-8e49-452f-be1b-8e0b3061323d admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:50.200634 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1536c070-8e49-452f-be1b-8e0b3061323d admin admin] PATH_INFO: `/v3/users/09e222ed698c4972812c47408d2745cd` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:50.202037 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1536c070-8e49-452f-be1b-8e0b3061323d admin admin] RBAC: Authorizing `identity:delete_user(user_id=09e222ed698c4972812c47408d2745cd)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:50.204294 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1536c070-8e49-452f-be1b-8e0b3061323d admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:50.248089 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdebbda8-0ffb-49e4-ad35-00e4fcbef947 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:50.249854 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdebbda8-0ffb-49e4-ad35-00e4fcbef947 service cinder] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:50.257961 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-1536c070-8e49-452f-be1b-8e0b3061323d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '09e222ed698c4972812c47408d2745cd'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:50.261390 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 993/1984] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:56:49 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 00:56:50.262300 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdebbda8-0ffb-49e4-ad35-00e4fcbef947 service cinder] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:50.276132 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdebbda8-0ffb-49e4-ad35-00e4fcbef947 service cinder] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:50.278346 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-1536c070-8e49-452f-be1b-8e0b3061323d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '09e222ed698c4972812c47408d2745cd'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:50.279300 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-1536c070-8e49-452f-be1b-8e0b3061323d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '09e222ed698c4972812c47408d2745cd'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:50.288843 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baaf58d9-6b8f-4743-b517-165d286ed706 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:50.291006 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baaf58d9-6b8f-4743-b517-165d286ed706 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:50.291006 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baaf58d9-6b8f-4743-b517-165d286ed706 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:50.291202 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baaf58d9-6b8f-4743-b517-165d286ed706 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:50.291771 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baaf58d9-6b8f-4743-b517-165d286ed706 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:50.295251 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-1536c070-8e49-452f-be1b-8e0b3061323d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '09e222ed698c4972812c47408d2745cd'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:50.295760 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baaf58d9-6b8f-4743-b517-165d286ed706 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:50.297434 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-1536c070-8e49-452f-be1b-8e0b3061323d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '09e222ed698c4972812c47408d2745cd'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:50.322867 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 994/1985] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:50 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:56:50.324161 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baaf58d9-6b8f-4743-b517-165d286ed706 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:50.328144 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-baaf58d9-6b8f-4743-b517-165d286ed706 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:50.338856 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 992/1986] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:56:50 2022] DELETE /identity/v3/users/09e222ed698c4972812c47408d2745cd => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:50.340325 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064cc122-8f58-4212-88c0-3189f54f1fcc None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:50.340602 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064cc122-8f58-4212-88c0-3189f54f1fcc None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:50.340867 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064cc122-8f58-4212-88c0-3189f54f1fcc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:50.353827 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baaf58d9-6b8f-4743-b517-165d286ed706 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:50.386207 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-064cc122-8f58-4212-88c0-3189f54f1fcc None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:50.387532 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-064cc122-8f58-4212-88c0-3189f54f1fcc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:50.482020 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a29d13-f098-4b52-9343-5fb2d5db2813 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:50.482644 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a29d13-f098-4b52-9343-5fb2d5db2813 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:50.482915 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a29d13-f098-4b52-9343-5fb2d5db2813 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:50.483180 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a29d13-f098-4b52-9343-5fb2d5db2813 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:50.483824 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a29d13-f098-4b52-9343-5fb2d5db2813 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:50.485485 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a29d13-f098-4b52-9343-5fb2d5db2813 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:50.486254 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 993/1987] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:50.487398 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-064cc122-8f58-4212-88c0-3189f54f1fcc None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:50.489481 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-064cc122-8f58-4212-88c0-3189f54f1fcc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:50.502448 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 995/1988] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:50 2022] GET /identity/v3/roles => generated 1420 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 00:56:50.507770 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a29d13-f098-4b52-9343-5fb2d5db2813 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:50.520983 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a29d13-f098-4b52-9343-5fb2d5db2813 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:50.526998 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f88319-7087-4250-98f0-2bf593a22fe0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:50.527659 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f88319-7087-4250-98f0-2bf593a22fe0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:50.527934 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f88319-7087-4250-98f0-2bf593a22fe0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:50.528204 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f88319-7087-4250-98f0-2bf593a22fe0 admin admin] PATH_INFO: `/v3/projects/e463fe4d96b04d6fa3224a417eeeeadc/users/0e9e011d2f014022968bed46bd10c2fb/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:50.550259 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93f88319-7087-4250-98f0-2bf593a22fe0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e463fe4d96b04d6fa3224a417eeeeadc, user_id=0e9e011d2f014022968bed46bd10c2fb, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:50.556048 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93f88319-7087-4250-98f0-2bf593a22fe0 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:50.585001 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 996/1989] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:56:50 2022] PUT /identity/v3/projects/e463fe4d96b04d6fa3224a417eeeeadc/users/0e9e011d2f014022968bed46bd10c2fb/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 76 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:50.587599 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f88319-7087-4250-98f0-2bf593a22fe0 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:50.593706 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-064cc122-8f58-4212-88c0-3189f54f1fcc None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:50.601665 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-93f88319-7087-4250-98f0-2bf593a22fe0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:50.657019 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:50.697817 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f106840-59bf-4e6d-acad-f480f3694e25 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:50.698444 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f106840-59bf-4e6d-acad-f480f3694e25 service cinder] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:50.698760 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f106840-59bf-4e6d-acad-f480f3694e25 service cinder] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:50.698972 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f106840-59bf-4e6d-acad-f480f3694e25 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:50.701634 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-2f106840-59bf-4e6d-acad-f480f3694e25 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:50.705009 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:50.765306 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f88319-7087-4250-98f0-2bf593a22fe0 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:50.775383 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a366be24-026e-48b2-ac9a-5f48ee8fb7d5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:50.776071 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a366be24-026e-48b2-ac9a-5f48ee8fb7d5 service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:50.776409 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a366be24-026e-48b2-ac9a-5f48ee8fb7d5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:50.776739 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a366be24-026e-48b2-ac9a-5f48ee8fb7d5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:50.806629 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f106840-59bf-4e6d-acad-f480f3694e25 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:50.808085 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f106840-59bf-4e6d-acad-f480f3694e25 service cinder] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:50.822809 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 994/1990] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:56:50 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 336 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:56:50.824488 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b8ffd7-453d-4aaa-b76c-9703c48c4144 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:50.824873 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b8ffd7-453d-4aaa-b76c-9703c48c4144 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:50.825443 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b8ffd7-453d-4aaa-b76c-9703c48c4144 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:50.849226 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-c6b8ffd7-453d-4aaa-b76c-9703c48c4144 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:50.851516 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-c6b8ffd7-453d-4aaa-b76c-9703c48c4144 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:50.915291 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a366be24-026e-48b2-ac9a-5f48ee8fb7d5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:50.917657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a366be24-026e-48b2-ac9a-5f48ee8fb7d5 service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:50.936864 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 997/1991] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:56:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 353 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:56:51.006398 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 995/1992] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:51.056874 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a366be24-026e-48b2-ac9a-5f48ee8fb7d5 service neutron] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:51.058496 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-a366be24-026e-48b2-ac9a-5f48ee8fb7d5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:51.074319 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a366be24-026e-48b2-ac9a-5f48ee8fb7d5 service neutron] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:51.088468 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57796272-af8c-4714-b2a1-5d3eaff74032 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:51.089182 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57796272-af8c-4714-b2a1-5d3eaff74032 service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:51.089524 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57796272-af8c-4714-b2a1-5d3eaff74032 service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:51.089999 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57796272-af8c-4714-b2a1-5d3eaff74032 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:51.114432 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57796272-af8c-4714-b2a1-5d3eaff74032 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:51.116040 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57796272-af8c-4714-b2a1-5d3eaff74032 service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:51.127672 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 998/1993] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:56:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 71 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:56:51.339861 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b8ffd7-453d-4aaa-b76c-9703c48c4144 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:51.341912 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-c6b8ffd7-453d-4aaa-b76c-9703c48c4144 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:51.368949 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b8ffd7-453d-4aaa-b76c-9703c48c4144 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:51.386017 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e6b8f48-f8fb-421e-87cc-d6b98a6fc719 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:51.388257 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6b8f48-f8fb-421e-87cc-d6b98a6fc719 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:51.390321 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6b8f48-f8fb-421e-87cc-d6b98a6fc719 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:51.391566 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6b8f48-f8fb-421e-87cc-d6b98a6fc719 admin admin] PATH_INFO: `/v3/users/c1cf6ec8b164449c8799da78a5f0a909` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:51.397438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e6b8f48-f8fb-421e-87cc-d6b98a6fc719 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c1cf6ec8b164449c8799da78a5f0a909)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:51.406367 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e6b8f48-f8fb-421e-87cc-d6b98a6fc719 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:51.593608 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-9e6b8f48-f8fb-421e-87cc-d6b98a6fc719 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c1cf6ec8b164449c8799da78a5f0a909'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:51.614084 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57796272-af8c-4714-b2a1-5d3eaff74032 service neutron] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:51.617214 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-57796272-af8c-4714-b2a1-5d3eaff74032 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:51.635159 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-9e6b8f48-f8fb-421e-87cc-d6b98a6fc719 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c1cf6ec8b164449c8799da78a5f0a909'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:51.638668 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-9e6b8f48-f8fb-421e-87cc-d6b98a6fc719 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c1cf6ec8b164449c8799da78a5f0a909'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:51.690880 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-9e6b8f48-f8fb-421e-87cc-d6b98a6fc719 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c1cf6ec8b164449c8799da78a5f0a909'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:51.695705 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-9e6b8f48-f8fb-421e-87cc-d6b98a6fc719 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c1cf6ec8b164449c8799da78a5f0a909'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:51.702199 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57796272-af8c-4714-b2a1-5d3eaff74032 service neutron] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:51.731766 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed0f688a-eebf-48ba-88a4-4fb96df5ac59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:51.734515 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0f688a-eebf-48ba-88a4-4fb96df5ac59 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:51.735316 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0f688a-eebf-48ba-88a4-4fb96df5ac59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:51.737185 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0f688a-eebf-48ba-88a4-4fb96df5ac59 admin admin] PATH_INFO: `/v3/projects/7cefb5d1dcb640a4b0ad87daa1418fae` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:51.743169 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:51.744545 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed0f688a-eebf-48ba-88a4-4fb96df5ac59 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7cefb5d1dcb640a4b0ad87daa1418fae)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:51.759764 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed0f688a-eebf-48ba-88a4-4fb96df5ac59 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:51.770826 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 996/1994] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:56:51 2022] DELETE /identity/v3/users/c1cf6ec8b164449c8799da78a5f0a909 => generated 0 bytes in 431 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:51.780024 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ab22cd-7d2d-4408-9fda-a8531c53a385 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:51.786421 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ab22cd-7d2d-4408-9fda-a8531c53a385 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:51.786986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ab22cd-7d2d-4408-9fda-a8531c53a385 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:51.860064 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-07ab22cd-7d2d-4408-9fda-a8531c53a385 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:51.862292 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-07ab22cd-7d2d-4408-9fda-a8531c53a385 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:51.940833 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-ed0f688a-eebf-48ba-88a4-4fb96df5ac59 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7cefb5d1dcb640a4b0ad87daa1418fae', 'request_id': 'req-ed0f688a-eebf-48ba-88a4-4fb96df5ac59'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:51.978610 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-ed0f688a-eebf-48ba-88a4-4fb96df5ac59 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7cefb5d1dcb640a4b0ad87daa1418fae', 'request_id': 'req-ed0f688a-eebf-48ba-88a4-4fb96df5ac59'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:52.006727 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-ed0f688a-eebf-48ba-88a4-4fb96df5ac59 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7cefb5d1dcb640a4b0ad87daa1418fae', 'request_id': 'req-ed0f688a-eebf-48ba-88a4-4fb96df5ac59'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:52.073653 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-ed0f688a-eebf-48ba-88a4-4fb96df5ac59 admin admin] The token cache is being invalidate because project 7cefb5d1dcb640a4b0ad87daa1418fae was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:56:52.075720 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-ed0f688a-eebf-48ba-88a4-4fb96df5ac59 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:52.078749 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-ed0f688a-eebf-48ba-88a4-4fb96df5ac59 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:52.088729 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 999/1995] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:56:51 2022] DELETE /identity/v3/projects/7cefb5d1dcb640a4b0ad87daa1418fae => generated 0 bytes in 471 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:52.091883 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed0f688a-eebf-48ba-88a4-4fb96df5ac59 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:52.094655 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-ed0f688a-eebf-48ba-88a4-4fb96df5ac59 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:52.159902 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 997/1996] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 377 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:52.267682 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ab22cd-7d2d-4408-9fda-a8531c53a385 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:52.274416 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-07ab22cd-7d2d-4408-9fda-a8531c53a385 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:52.303971 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:52.370445 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed0f688a-eebf-48ba-88a4-4fb96df5ac59 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:52.394291 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6152145a-88f8-4bd8-9c0c-b43a4c89d654 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:52.395767 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6152145a-88f8-4bd8-9c0c-b43a4c89d654 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:52.397950 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6152145a-88f8-4bd8-9c0c-b43a4c89d654 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:52.401137 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6152145a-88f8-4bd8-9c0c-b43a4c89d654 admin admin] PATH_INFO: `/v3/users/8c48b627d65c470f83b259720f83b106` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:52.406375 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6152145a-88f8-4bd8-9c0c-b43a4c89d654 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8c48b627d65c470f83b259720f83b106)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:52.411817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6152145a-88f8-4bd8-9c0c-b43a4c89d654 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:52.542938 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ab22cd-7d2d-4408-9fda-a8531c53a385 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:52.556420 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-6152145a-88f8-4bd8-9c0c-b43a4c89d654 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8c48b627d65c470f83b259720f83b106'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:52.561876 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-6152145a-88f8-4bd8-9c0c-b43a4c89d654 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8c48b627d65c470f83b259720f83b106'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:52.576570 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-6152145a-88f8-4bd8-9c0c-b43a4c89d654 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8c48b627d65c470f83b259720f83b106'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:52.578316 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-6152145a-88f8-4bd8-9c0c-b43a4c89d654 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8c48b627d65c470f83b259720f83b106'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:52.592038 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-6152145a-88f8-4bd8-9c0c-b43a4c89d654 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8c48b627d65c470f83b259720f83b106'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:52.635132 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1000/1997] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:56:52 2022] DELETE /identity/v3/users/8c48b627d65c470f83b259720f83b106 => generated 0 bytes in 541 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 00:56:52.642124 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c060f8e4-5813-41de-93f3-518477f1fc11 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:52.642929 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c060f8e4-5813-41de-93f3-518477f1fc11 service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:52.643204 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c060f8e4-5813-41de-93f3-518477f1fc11 service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:52.643525 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c060f8e4-5813-41de-93f3-518477f1fc11 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:52.645359 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-c060f8e4-5813-41de-93f3-518477f1fc11 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:52.661564 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c060f8e4-5813-41de-93f3-518477f1fc11 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:52.664110 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c060f8e4-5813-41de-93f3-518477f1fc11 service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:52.676058 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 998/1998] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:56:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 410 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 00:56:52.970428 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2c4e96-b081-4354-89e7-a467b206bcfd None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:52.970716 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2c4e96-b081-4354-89e7-a467b206bcfd None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:52.971091 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2c4e96-b081-4354-89e7-a467b206bcfd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:53.017644 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-0d2c4e96-b081-4354-89e7-a467b206bcfd None None] MFA Rules not processed for user `0e9e011d2f014022968bed46bd10c2fb`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:53.020133 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-0d2c4e96-b081-4354-89e7-a467b206bcfd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:53.123685 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1001/1999] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:52 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:53.134919 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c060f8e4-5813-41de-93f3-518477f1fc11 service neutron] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:53.144296 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-c060f8e4-5813-41de-93f3-518477f1fc11 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:53.163158 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c060f8e4-5813-41de-93f3-518477f1fc11 service neutron] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:53.242367 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:DELETE=1,keystone:SELECT=99 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:53.275893 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce347dcc-7a3e-4845-9feb-caa6eec2fed9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:53.276698 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce347dcc-7a3e-4845-9feb-caa6eec2fed9 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:53.277039 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce347dcc-7a3e-4845-9feb-caa6eec2fed9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:53.277406 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce347dcc-7a3e-4845-9feb-caa6eec2fed9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:53.278138 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce347dcc-7a3e-4845-9feb-caa6eec2fed9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:53.280317 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce347dcc-7a3e-4845-9feb-caa6eec2fed9 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:53.314115 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d2c4e96-b081-4354-89e7-a467b206bcfd None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:53.316126 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-0d2c4e96-b081-4354-89e7-a467b206bcfd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:53.356174 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d2c4e96-b081-4354-89e7-a467b206bcfd None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:53.371421 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce6502c-5d0f-4881-9874-e36fb7aa9fcf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:53.372791 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce6502c-5d0f-4881-9874-e36fb7aa9fcf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:53.373150 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce6502c-5d0f-4881-9874-e36fb7aa9fcf admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:53.373567 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce6502c-5d0f-4881-9874-e36fb7aa9fcf admin admin] PATH_INFO: `/v3/projects/949e5c4b9d274f488df36e640d3b9eec` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:53.373718 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 999/2000] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:56:53 2022] POST /identity/v3/projects => generated 373 bytes in 240 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:56:53.381739 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce347dcc-7a3e-4845-9feb-caa6eec2fed9 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:53.399805 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ce6502c-5d0f-4881-9874-e36fb7aa9fcf admin admin] RBAC: Authorizing `identity:delete_project(project_id=949e5c4b9d274f488df36e640d3b9eec)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:53.403689 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ce6502c-5d0f-4881-9874-e36fb7aa9fcf admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:53.626533 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-5ce6502c-5d0f-4881-9874-e36fb7aa9fcf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '949e5c4b9d274f488df36e640d3b9eec', 'request_id': 'req-5ce6502c-5d0f-4881-9874-e36fb7aa9fcf'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:53.648579 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-5ce6502c-5d0f-4881-9874-e36fb7aa9fcf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '949e5c4b9d274f488df36e640d3b9eec', 'request_id': 'req-5ce6502c-5d0f-4881-9874-e36fb7aa9fcf'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:53.650726 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:53.677839 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-5ce6502c-5d0f-4881-9874-e36fb7aa9fcf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '949e5c4b9d274f488df36e640d3b9eec', 'request_id': 'req-5ce6502c-5d0f-4881-9874-e36fb7aa9fcf'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:53.699326 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-5ce6502c-5d0f-4881-9874-e36fb7aa9fcf admin admin] The token cache is being invalidate because project 949e5c4b9d274f488df36e640d3b9eec was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:56:53.699326 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-5ce6502c-5d0f-4881-9874-e36fb7aa9fcf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:53.700811 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-5ce6502c-5d0f-4881-9874-e36fb7aa9fcf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:53.715523 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce347dcc-7a3e-4845-9feb-caa6eec2fed9 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:53.720627 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1002/2001] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:56:53 2022] DELETE /identity/v3/projects/949e5c4b9d274f488df36e640d3b9eec => generated 0 bytes in 407 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:53.721657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce6502c-5d0f-4881-9874-e36fb7aa9fcf admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:53.733963 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-5ce6502c-5d0f-4881-9874-e36fb7aa9fcf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:53.854979 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce6502c-5d0f-4881-9874-e36fb7aa9fcf admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:53.857445 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0d7b63-549f-4d5f-9c78-89fa714c9298 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:53.858384 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0d7b63-549f-4d5f-9c78-89fa714c9298 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:53.858733 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0d7b63-549f-4d5f-9c78-89fa714c9298 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:53.859461 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0d7b63-549f-4d5f-9c78-89fa714c9298 admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:53.860478 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee0d7b63-549f-4d5f-9c78-89fa714c9298 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:53.863054 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee0d7b63-549f-4d5f-9c78-89fa714c9298 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:53.863859 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5837163d-f764-40b1-a60a-dd712b559b17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:53.865790 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5837163d-f764-40b1-a60a-dd712b559b17 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:53.866233 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5837163d-f764-40b1-a60a-dd712b559b17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:53.866902 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5837163d-f764-40b1-a60a-dd712b559b17 admin admin] PATH_INFO: `/v3/projects/ebcbfdd554ab41299a6932ab451d0072` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:53.868362 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5837163d-f764-40b1-a60a-dd712b559b17 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ebcbfdd554ab41299a6932ab451d0072)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:53.870846 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5837163d-f764-40b1-a60a-dd712b559b17 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:53.900213 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1000/2002] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:56:53 2022] POST /identity/v3/users => generated 293 bytes in 520 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:56:53.902530 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0d7b63-549f-4d5f-9c78-89fa714c9298 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:53.909850 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-ee0d7b63-549f-4d5f-9c78-89fa714c9298 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:53.938184 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0d7b63-549f-4d5f-9c78-89fa714c9298 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:53.963645 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-5837163d-f764-40b1-a60a-dd712b559b17 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ebcbfdd554ab41299a6932ab451d0072', 'request_id': 'req-5837163d-f764-40b1-a60a-dd712b559b17'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:53.988268 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-5837163d-f764-40b1-a60a-dd712b559b17 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ebcbfdd554ab41299a6932ab451d0072', 'request_id': 'req-5837163d-f764-40b1-a60a-dd712b559b17'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:54.006080 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-5837163d-f764-40b1-a60a-dd712b559b17 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ebcbfdd554ab41299a6932ab451d0072', 'request_id': 'req-5837163d-f764-40b1-a60a-dd712b559b17'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:54.029363 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-5837163d-f764-40b1-a60a-dd712b559b17 admin admin] The token cache is being invalidate because project ebcbfdd554ab41299a6932ab451d0072 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:56:54.030326 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-5837163d-f764-40b1-a60a-dd712b559b17 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:54.031338 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-5837163d-f764-40b1-a60a-dd712b559b17 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:54.033290 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1003/2003] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:56:53 2022] DELETE /identity/v3/projects/ebcbfdd554ab41299a6932ab451d0072 => generated 0 bytes in 312 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:54.034420 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5837163d-f764-40b1-a60a-dd712b559b17 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:54.039374 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-5837163d-f764-40b1-a60a-dd712b559b17 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:54.055144 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e6a9e2-c7cb-4c79-8f9b-e5eb777f5a8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:54.055883 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e6a9e2-c7cb-4c79-8f9b-e5eb777f5a8c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:54.056059 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e6a9e2-c7cb-4c79-8f9b-e5eb777f5a8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:54.056306 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e6a9e2-c7cb-4c79-8f9b-e5eb777f5a8c admin admin] PATH_INFO: `/v3/users/088623bec3a64727add0c5f1729241d7` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:54.057655 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3e6a9e2-c7cb-4c79-8f9b-e5eb777f5a8c admin admin] RBAC: Authorizing `identity:delete_user(user_id=088623bec3a64727add0c5f1729241d7)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:54.059614 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3e6a9e2-c7cb-4c79-8f9b-e5eb777f5a8c admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:54.092155 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=95,keystone:INSERT=4,keystone:DELETE=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:54.126954 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-a3e6a9e2-c7cb-4c79-8f9b-e5eb777f5a8c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '088623bec3a64727add0c5f1729241d7'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:54.137080 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-a3e6a9e2-c7cb-4c79-8f9b-e5eb777f5a8c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '088623bec3a64727add0c5f1729241d7'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:54.138292 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-a3e6a9e2-c7cb-4c79-8f9b-e5eb777f5a8c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '088623bec3a64727add0c5f1729241d7'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:54.141566 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5837163d-f764-40b1-a60a-dd712b559b17 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:54.147033 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-a3e6a9e2-c7cb-4c79-8f9b-e5eb777f5a8c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '088623bec3a64727add0c5f1729241d7'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:54.147564 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-a3e6a9e2-c7cb-4c79-8f9b-e5eb777f5a8c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '088623bec3a64727add0c5f1729241d7'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:54.149138 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1246e54-c7e2-410c-bf04-1a07db3cc89d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:54.149848 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1246e54-c7e2-410c-bf04-1a07db3cc89d admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:54.150188 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1246e54-c7e2-410c-bf04-1a07db3cc89d admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:54.150522 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1246e54-c7e2-410c-bf04-1a07db3cc89d admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:54.151222 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1246e54-c7e2-410c-bf04-1a07db3cc89d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:54.152821 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1246e54-c7e2-410c-bf04-1a07db3cc89d admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:54.163893 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1004/2004] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:54 2022] GET /identity/v3/roles => generated 1420 bytes in 130 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:56:54.165398 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8650024-1186-4cb2-b1d9-1734292331f7 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:54.165683 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8650024-1186-4cb2-b1d9-1734292331f7 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:54.165956 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8650024-1186-4cb2-b1d9-1734292331f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:54.182876 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1001/2005] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:56:53 2022] DELETE /identity/v3/users/088623bec3a64727add0c5f1729241d7 => generated 0 bytes in 282 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:54.183831 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e6a9e2-c7cb-4c79-8f9b-e5eb777f5a8c admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:54.185365 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-a3e6a9e2-c7cb-4c79-8f9b-e5eb777f5a8c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:54.188530 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-e8650024-1186-4cb2-b1d9-1734292331f7 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:54.189771 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-e8650024-1186-4cb2-b1d9-1734292331f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:54.197954 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:DELETE=9,keystone:SELECT=90,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:54.286796 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1005/2006] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:54.295085 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c663596a-3cd8-4604-bd10-938eebe781eb None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:54.295380 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c663596a-3cd8-4604-bd10-938eebe781eb None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:54.295671 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c663596a-3cd8-4604-bd10-938eebe781eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:54.322948 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e6a9e2-c7cb-4c79-8f9b-e5eb777f5a8c admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:54.325348 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-c663596a-3cd8-4604-bd10-938eebe781eb None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:54.328286 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-c663596a-3cd8-4604-bd10-938eebe781eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:54.338286 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ae3cf7-62b7-46db-b87b-73e6840579f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:54.339672 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ae3cf7-62b7-46db-b87b-73e6840579f4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:54.339824 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1006/2007] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 00:56:54.340428 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ae3cf7-62b7-46db-b87b-73e6840579f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:54.340933 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ae3cf7-62b7-46db-b87b-73e6840579f4 admin admin] PATH_INFO: `/v3/projects/254d347551d748c3b8356d11a8e309c1/users/ea301e7372b841cba1ebf8c81699c230/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:54.344140 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c663596a-3cd8-4604-bd10-938eebe781eb None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:54.361958 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c663596a-3cd8-4604-bd10-938eebe781eb None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:54.382462 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06d1e7df-74cb-45b1-bf1c-840c6c59a286 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:54.383334 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d1e7df-74cb-45b1-bf1c-840c6c59a286 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:54.383687 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d1e7df-74cb-45b1-bf1c-840c6c59a286 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:54.384516 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d1e7df-74cb-45b1-bf1c-840c6c59a286 admin admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:54.397551 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ae3cf7-62b7-46db-b87b-73e6840579f4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=254d347551d748c3b8356d11a8e309c1, user_id=ea301e7372b841cba1ebf8c81699c230, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:54.399885 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06d1e7df-74cb-45b1-bf1c-840c6c59a286 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:56:54.400285 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06d1e7df-74cb-45b1-bf1c-840c6c59a286 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:54.403194 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ae3cf7-62b7-46db-b87b-73e6840579f4 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:54.407003 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06d1e7df-74cb-45b1-bf1c-840c6c59a286 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:54.434189 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1007/2008] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 00:56:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:56:54.438163 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06d1e7df-74cb-45b1-bf1c-840c6c59a286 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:54.441455 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1002/2009] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:56:54 2022] PUT /identity/v3/projects/254d347551d748c3b8356d11a8e309c1/users/ea301e7372b841cba1ebf8c81699c230/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 258 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:54.446068 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ae3cf7-62b7-46db-b87b-73e6840579f4 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:54.455817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06d1e7df-74cb-45b1-bf1c-840c6c59a286 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:54.539814 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b563015e-1eb5-44a2-8cd0-bcbd9c8da38a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:54.540518 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b563015e-1eb5-44a2-8cd0-bcbd9c8da38a admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:54.541022 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b563015e-1eb5-44a2-8cd0-bcbd9c8da38a admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:54.541488 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b563015e-1eb5-44a2-8cd0-bcbd9c8da38a admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:54.542128 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b563015e-1eb5-44a2-8cd0-bcbd9c8da38a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:54.544069 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b563015e-1eb5-44a2-8cd0-bcbd9c8da38a admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:54.553308 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ae3cf7-62b7-46db-b87b-73e6840579f4 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:54.558131 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7848e5-41e8-464b-a274-cb955016b82f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:54.558602 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7848e5-41e8-464b-a274-cb955016b82f admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:54.558883 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7848e5-41e8-464b-a274-cb955016b82f admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:54.559129 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7848e5-41e8-464b-a274-cb955016b82f admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:54.559776 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f7848e5-41e8-464b-a274-cb955016b82f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:54.562061 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f7848e5-41e8-464b-a274-cb955016b82f admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:54.576916 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1003/2010] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:54 2022] GET /identity/v3/roles => generated 1420 bytes in 128 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:56:54.577848 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1008/2011] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:56:54 2022] POST /identity/v3/projects => generated 385 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:56:54.581168 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7848e5-41e8-464b-a274-cb955016b82f admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:54.582139 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b563015e-1eb5-44a2-8cd0-bcbd9c8da38a admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:54.595899 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7848e5-41e8-464b-a274-cb955016b82f admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:54.607791 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:DELETE=9,keystone:SELECT=89,keystone:INSERT=2 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:54.665547 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b563015e-1eb5-44a2-8cd0-bcbd9c8da38a admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:54.670502 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5178ab47-e3c9-494d-95c8-4982f1ea62a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:54.671219 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5178ab47-e3c9-494d-95c8-4982f1ea62a2 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:54.671434 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5178ab47-e3c9-494d-95c8-4982f1ea62a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:54.671702 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5178ab47-e3c9-494d-95c8-4982f1ea62a2 admin admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:54.672349 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5178ab47-e3c9-494d-95c8-4982f1ea62a2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:54.673480 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd0d0ef6-4e71-4f3f-b332-1f8393f5f9df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:54.674261 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd0d0ef6-4e71-4f3f-b332-1f8393f5f9df admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:54.674372 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5178ab47-e3c9-494d-95c8-4982f1ea62a2 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:54.674622 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd0d0ef6-4e71-4f3f-b332-1f8393f5f9df admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:54.674969 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd0d0ef6-4e71-4f3f-b332-1f8393f5f9df admin admin] PATH_INFO: `/v3/projects/254d347551d748c3b8356d11a8e309c1/users/ea301e7372b841cba1ebf8c81699c230/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:54.680356 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd0d0ef6-4e71-4f3f-b332-1f8393f5f9df admin admin] RBAC: Authorizing `identity:create_grant(project_id=254d347551d748c3b8356d11a8e309c1, user_id=ea301e7372b841cba1ebf8c81699c230, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:54.683992 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd0d0ef6-4e71-4f3f-b332-1f8393f5f9df admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:54.696180 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1004/2012] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:56:54 2022] PUT /identity/v3/projects/254d347551d748c3b8356d11a8e309c1/users/ea301e7372b841cba1ebf8c81699c230/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:54.697178 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd0d0ef6-4e71-4f3f-b332-1f8393f5f9df admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:54.701104 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-dd0d0ef6-4e71-4f3f-b332-1f8393f5f9df admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:54.703070 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1009/2013] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:56:54 2022] POST /identity/v3/users => generated 293 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 00:56:54.706920 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5178ab47-e3c9-494d-95c8-4982f1ea62a2 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:54.723047 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5178ab47-e3c9-494d-95c8-4982f1ea62a2 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:54.729297 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:54.808758 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd0d0ef6-4e71-4f3f-b332-1f8393f5f9df admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:54.814121 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f64ee2-b4c1-407a-bf4b-b40877613d33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:54.814759 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f64ee2-b4c1-407a-bf4b-b40877613d33 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:54.815039 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f64ee2-b4c1-407a-bf4b-b40877613d33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:54.815358 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f64ee2-b4c1-407a-bf4b-b40877613d33 admin admin] PATH_INFO: `/v3/projects/56d0ea5794f14f8d96f6b48d16fa108e` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:54.816022 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3bba5d6-c0ae-48f5-a235-84f8f6e5da14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:54.816472 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61f64ee2-b4c1-407a-bf4b-b40877613d33 admin admin] RBAC: Authorizing `identity:delete_project(project_id=56d0ea5794f14f8d96f6b48d16fa108e)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:54.818730 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61f64ee2-b4c1-407a-bf4b-b40877613d33 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:54.819098 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3bba5d6-c0ae-48f5-a235-84f8f6e5da14 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:54.819407 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3bba5d6-c0ae-48f5-a235-84f8f6e5da14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:54.819683 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3bba5d6-c0ae-48f5-a235-84f8f6e5da14 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:54.820349 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3bba5d6-c0ae-48f5-a235-84f8f6e5da14 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:54.822071 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3bba5d6-c0ae-48f5-a235-84f8f6e5da14 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:54.833386 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1010/2014] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:54 2022] GET /identity/v3/roles => generated 1420 bytes in 127 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 00:56:54.834267 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3bba5d6-c0ae-48f5-a235-84f8f6e5da14 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:54.835996 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-c3bba5d6-c0ae-48f5-a235-84f8f6e5da14 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:54.851680 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3bba5d6-c0ae-48f5-a235-84f8f6e5da14 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:54.858782 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f885e5-74e7-4f17-9d07-74442bd5c968 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:54.859486 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f885e5-74e7-4f17-9d07-74442bd5c968 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:54.859764 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f885e5-74e7-4f17-9d07-74442bd5c968 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:54.860056 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f885e5-74e7-4f17-9d07-74442bd5c968 admin admin] PATH_INFO: `/v3/users/a5dfd7e712454d7db264058178c6bb7c` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:54.861236 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55f885e5-74e7-4f17-9d07-74442bd5c968 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a5dfd7e712454d7db264058178c6bb7c)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:54.863470 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55f885e5-74e7-4f17-9d07-74442bd5c968 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:54.885999 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-61f64ee2-b4c1-407a-bf4b-b40877613d33 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '56d0ea5794f14f8d96f6b48d16fa108e', 'request_id': 'req-61f64ee2-b4c1-407a-bf4b-b40877613d33'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:54.898082 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-61f64ee2-b4c1-407a-bf4b-b40877613d33 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '56d0ea5794f14f8d96f6b48d16fa108e', 'request_id': 'req-61f64ee2-b4c1-407a-bf4b-b40877613d33'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:54.906227 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-61f64ee2-b4c1-407a-bf4b-b40877613d33 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '56d0ea5794f14f8d96f6b48d16fa108e', 'request_id': 'req-61f64ee2-b4c1-407a-bf4b-b40877613d33'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:54.923621 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-55f885e5-74e7-4f17-9d07-74442bd5c968 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a5dfd7e712454d7db264058178c6bb7c'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:54.924708 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-61f64ee2-b4c1-407a-bf4b-b40877613d33 admin admin] The token cache is being invalidate because project 56d0ea5794f14f8d96f6b48d16fa108e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81386) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:56:54.924901 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-55f885e5-74e7-4f17-9d07-74442bd5c968 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a5dfd7e712454d7db264058178c6bb7c'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:54.925017 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-61f64ee2-b4c1-407a-bf4b-b40877613d33 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:54.925796 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-61f64ee2-b4c1-407a-bf4b-b40877613d33 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:54.936842 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1005/2015] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:56:54 2022] DELETE /identity/v3/projects/56d0ea5794f14f8d96f6b48d16fa108e => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:54.936842 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f64ee2-b4c1-407a-bf4b-b40877613d33 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:54.936842 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-61f64ee2-b4c1-407a-bf4b-b40877613d33 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:54.944937 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-55f885e5-74e7-4f17-9d07-74442bd5c968 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a5dfd7e712454d7db264058178c6bb7c'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:54.945776 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-55f885e5-74e7-4f17-9d07-74442bd5c968 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a5dfd7e712454d7db264058178c6bb7c'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:54.955927 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-55f885e5-74e7-4f17-9d07-74442bd5c968 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a5dfd7e712454d7db264058178c6bb7c'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:54.980082 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1011/2016] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:56:54 2022] DELETE /identity/v3/users/a5dfd7e712454d7db264058178c6bb7c => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:54.982207 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1abcc25-2719-4588-b3bd-ac23ff250ed8 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:54.982534 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1abcc25-2719-4588-b3bd-ac23ff250ed8 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:54.982808 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1abcc25-2719-4588-b3bd-ac23ff250ed8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:55.009286 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:55.024843 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-b1abcc25-2719-4588-b3bd-ac23ff250ed8 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:55.026109 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-b1abcc25-2719-4588-b3bd-ac23ff250ed8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:55.033242 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f64ee2-b4c1-407a-bf4b-b40877613d33 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:55.109350 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1012/2017] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:55.110535 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b57d7d9-7997-4c8e-bdae-bf874ab1c853 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:55.110970 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b57d7d9-7997-4c8e-bdae-bf874ab1c853 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:55.111249 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b57d7d9-7997-4c8e-bdae-bf874ab1c853 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:55.118791 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34894188-3a6b-4a9b-90c7-5586c1b5aefd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:55.119632 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34894188-3a6b-4a9b-90c7-5586c1b5aefd admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:55.120405 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34894188-3a6b-4a9b-90c7-5586c1b5aefd admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:55.120405 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34894188-3a6b-4a9b-90c7-5586c1b5aefd admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:55.121804 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34894188-3a6b-4a9b-90c7-5586c1b5aefd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:55.124327 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34894188-3a6b-4a9b-90c7-5586c1b5aefd admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:55.141164 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1006/2018] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:54 2022] GET /identity/v3/roles => generated 1420 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:56:55.143352 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092a5c12-3aac-4dc9-8887-ac48e1dea2c6 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:55.143818 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092a5c12-3aac-4dc9-8887-ac48e1dea2c6 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:55.144238 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092a5c12-3aac-4dc9-8887-ac48e1dea2c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:55.144932 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-9b57d7d9-7997-4c8e-bdae-bf874ab1c853 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:55.150220 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-9b57d7d9-7997-4c8e-bdae-bf874ab1c853 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:55.172208 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1013/2019] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 63 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:55.177765 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b57d7d9-7997-4c8e-bdae-bf874ab1c853 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:55.181232 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-9b57d7d9-7997-4c8e-bdae-bf874ab1c853 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:55.196211 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-092a5c12-3aac-4dc9-8887-ac48e1dea2c6 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:55.199138 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-092a5c12-3aac-4dc9-8887-ac48e1dea2c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:55.213533 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b57d7d9-7997-4c8e-bdae-bf874ab1c853 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:55.226229 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1007/2020] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:55.226694 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-092a5c12-3aac-4dc9-8887-ac48e1dea2c6 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:55.228963 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-092a5c12-3aac-4dc9-8887-ac48e1dea2c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:55.239331 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e02b81-b76c-4ccd-b945-eb80ac1a2f06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:55.240124 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e02b81-b76c-4ccd-b945-eb80ac1a2f06 admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:55.240569 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e02b81-b76c-4ccd-b945-eb80ac1a2f06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:55.241293 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e02b81-b76c-4ccd-b945-eb80ac1a2f06 admin admin] PATH_INFO: `/v3/projects/7b7dff005bcf45ce9fb79fdc2b8eab52/users/901ee7990f2c46eb913fab6b6f180e76/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:55.246287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:55.272237 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2e02b81-b76c-4ccd-b945-eb80ac1a2f06 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7b7dff005bcf45ce9fb79fdc2b8eab52, user_id=901ee7990f2c46eb913fab6b6f180e76, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:55.282416 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2e02b81-b76c-4ccd-b945-eb80ac1a2f06 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:55.311512 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1014/2021] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:56:55 2022] PUT /identity/v3/projects/7b7dff005bcf45ce9fb79fdc2b8eab52/users/901ee7990f2c46eb913fab6b6f180e76/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:55.325998 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e02b81-b76c-4ccd-b945-eb80ac1a2f06 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:55.371200 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-092a5c12-3aac-4dc9-8887-ac48e1dea2c6 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:55.451101 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e02b81-b76c-4ccd-b945-eb80ac1a2f06 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:55.457189 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc244885-1df7-4872-a10a-cc962229c162 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:55.458037 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc244885-1df7-4872-a10a-cc962229c162 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:55.458321 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc244885-1df7-4872-a10a-cc962229c162 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:55.458691 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc244885-1df7-4872-a10a-cc962229c162 admin admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:55.459644 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc244885-1df7-4872-a10a-cc962229c162 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:56:55.460155 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc244885-1df7-4872-a10a-cc962229c162 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:55.462156 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc244885-1df7-4872-a10a-cc962229c162 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:55.462951 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b4060d-1b34-4fc7-9a0a-77cf1727ac3a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:55.464525 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b4060d-1b34-4fc7-9a0a-77cf1727ac3a service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:55.464945 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b4060d-1b34-4fc7-9a0a-77cf1727ac3a service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:55.466517 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b4060d-1b34-4fc7-9a0a-77cf1727ac3a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:55.478484 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1015/2022] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 00:56:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:56:55.480800 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f81daa5d-1ec0-48d8-a598-848f1fb9417a None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:55.481400 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f81daa5d-1ec0-48d8-a598-848f1fb9417a None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:55.481540 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f81daa5d-1ec0-48d8-a598-848f1fb9417a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:55.494245 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6b4060d-1b34-4fc7-9a0a-77cf1727ac3a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:55.496076 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6b4060d-1b34-4fc7-9a0a-77cf1727ac3a service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:55.538868 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1008/2023] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:56:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:56:55.539972 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b4060d-1b34-4fc7-9a0a-77cf1727ac3a service neutron] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:55.540456 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-f81daa5d-1ec0-48d8-a598-848f1fb9417a None None] MFA Rules not processed for user `901ee7990f2c46eb913fab6b6f180e76`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:55.542106 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-f81daa5d-1ec0-48d8-a598-848f1fb9417a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:55.564203 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b4060d-1b34-4fc7-9a0a-77cf1727ac3a service neutron] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:55.580073 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75472b85-08db-415b-aefe-f77513d17906 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:55.580972 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75472b85-08db-415b-aefe-f77513d17906 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:55.581425 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75472b85-08db-415b-aefe-f77513d17906 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:55.581837 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75472b85-08db-415b-aefe-f77513d17906 admin admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:55.582655 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75472b85-08db-415b-aefe-f77513d17906 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:55.585533 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75472b85-08db-415b-aefe-f77513d17906 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:55.627340 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1009/2024] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:56:55 2022] POST /identity/v3/projects => generated 375 bytes in 87 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:56:55.631162 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75472b85-08db-415b-aefe-f77513d17906 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:55.653910 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:55.657997 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1016/2025] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:55 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:55.664158 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f81daa5d-1ec0-48d8-a598-848f1fb9417a None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:55.681941 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f81daa5d-1ec0-48d8-a598-848f1fb9417a None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:55.754141 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75472b85-08db-415b-aefe-f77513d17906 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:55.754701 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f933520b-ceea-4ad9-be5f-9f942a297be7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:55.755715 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f933520b-ceea-4ad9-be5f-9f942a297be7 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:55.756389 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f933520b-ceea-4ad9-be5f-9f942a297be7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:55.757099 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f933520b-ceea-4ad9-be5f-9f942a297be7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:55.758342 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f933520b-ceea-4ad9-be5f-9f942a297be7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:55.761221 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f933520b-ceea-4ad9-be5f-9f942a297be7 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:55.768631 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7285e412-b93b-4bbe-ae44-1c75959ee879 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:55.770334 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7285e412-b93b-4bbe-ae44-1c75959ee879 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:55.770623 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7285e412-b93b-4bbe-ae44-1c75959ee879 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:55.770898 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7285e412-b93b-4bbe-ae44-1c75959ee879 admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:55.772208 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7285e412-b93b-4bbe-ae44-1c75959ee879 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:55.783094 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7285e412-b93b-4bbe-ae44-1c75959ee879 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:55.784181 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:55.828969 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1017/2026] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:56:55 2022] POST /identity/v3/projects => generated 385 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:56:55.832988 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f933520b-ceea-4ad9-be5f-9f942a297be7 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:55.846357 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1010/2027] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:56:55 2022] POST /identity/v3/users => generated 288 bytes in 216 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:56:55.850985 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7285e412-b93b-4bbe-ae44-1c75959ee879 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:55.875055 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7285e412-b93b-4bbe-ae44-1c75959ee879 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:55.945509 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f933520b-ceea-4ad9-be5f-9f942a297be7 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:55.950584 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af58e5b-dc2b-4574-b5d8-8144e6c93a08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:55.951379 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af58e5b-dc2b-4574-b5d8-8144e6c93a08 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:55.951764 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af58e5b-dc2b-4574-b5d8-8144e6c93a08 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:55.952140 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af58e5b-dc2b-4574-b5d8-8144e6c93a08 admin admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:55.952879 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6af58e5b-dc2b-4574-b5d8-8144e6c93a08 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:55.954950 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6af58e5b-dc2b-4574-b5d8-8144e6c93a08 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:55.974068 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e80a735-3bdd-4932-9a41-59819f7efdd5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:55.975422 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e80a735-3bdd-4932-9a41-59819f7efdd5 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:55.975738 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e80a735-3bdd-4932-9a41-59819f7efdd5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:55.976007 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e80a735-3bdd-4932-9a41-59819f7efdd5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:55.976848 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e80a735-3bdd-4932-9a41-59819f7efdd5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:55.979774 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e80a735-3bdd-4932-9a41-59819f7efdd5 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:55.995109 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1011/2028] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:55 2022] GET /identity/v3/roles => generated 1420 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:56:55.996206 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e80a735-3bdd-4932-9a41-59819f7efdd5 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:56.001292 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1018/2029] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:56:55 2022] POST /identity/v3/users => generated 299 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:56:56.002433 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af58e5b-dc2b-4574-b5d8-8144e6c93a08 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:56.042555 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e80a735-3bdd-4932-9a41-59819f7efdd5 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:56.049782 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af58e5b-dc2b-4574-b5d8-8144e6c93a08 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:56.055070 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0b71641-ef1d-4a52-8754-87065068e7c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:56.059088 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b71641-ef1d-4a52-8754-87065068e7c4 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:56.059634 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b71641-ef1d-4a52-8754-87065068e7c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:56.060110 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b71641-ef1d-4a52-8754-87065068e7c4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:56.063168 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0b71641-ef1d-4a52-8754-87065068e7c4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:56.075165 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0b71641-ef1d-4a52-8754-87065068e7c4 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:56.075650 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d026bf0-42d7-4020-a04c-9ed977041b4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:56.077426 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d026bf0-42d7-4020-a04c-9ed977041b4a admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:56.077722 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d026bf0-42d7-4020-a04c-9ed977041b4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:56.077995 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d026bf0-42d7-4020-a04c-9ed977041b4a admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:56.078671 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d026bf0-42d7-4020-a04c-9ed977041b4a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:56.082485 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d026bf0-42d7-4020-a04c-9ed977041b4a admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:56.092837 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1012/2030] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:55 2022] GET /identity/v3/roles => generated 1420 bytes in 97 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:56:56.094356 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8902112d-98fa-4552-8fff-433d96d09b9d None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:56.096375 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8902112d-98fa-4552-8fff-433d96d09b9d None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:56.096725 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8902112d-98fa-4552-8fff-433d96d09b9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:56.110582 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1019/2031] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:56 2022] GET /identity/v3/roles => generated 1420 bytes in 109 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 00:56:56.113513 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d026bf0-42d7-4020-a04c-9ed977041b4a admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:56.131154 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d026bf0-42d7-4020-a04c-9ed977041b4a admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:56.136624 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18ad5aa-6dfe-41ee-a3a9-f7dee325f231 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:56.137351 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18ad5aa-6dfe-41ee-a3a9-f7dee325f231 admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:56.138522 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18ad5aa-6dfe-41ee-a3a9-f7dee325f231 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:56.138801 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18ad5aa-6dfe-41ee-a3a9-f7dee325f231 admin admin] PATH_INFO: `/v3/projects/ba896797e1ca468dbfaeca0b414589c8/users/e8c56fbde6844c45b487a81159500970/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:56.152766 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-8902112d-98fa-4552-8fff-433d96d09b9d None None] MFA Rules not processed for user `ea301e7372b841cba1ebf8c81699c230`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:56.154108 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-8902112d-98fa-4552-8fff-433d96d09b9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:56.159374 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b18ad5aa-6dfe-41ee-a3a9-f7dee325f231 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ba896797e1ca468dbfaeca0b414589c8, user_id=e8c56fbde6844c45b487a81159500970, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:56.162986 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b18ad5aa-6dfe-41ee-a3a9-f7dee325f231 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:56.180254 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1020/2032] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:56:56 2022] PUT /identity/v3/projects/ba896797e1ca468dbfaeca0b414589c8/users/e8c56fbde6844c45b487a81159500970/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:56.181101 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18ad5aa-6dfe-41ee-a3a9-f7dee325f231 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:56.267257 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18ad5aa-6dfe-41ee-a3a9-f7dee325f231 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:56.267731 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1013/2033] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:56 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:56.269188 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2564962-f49c-4b41-85f7-654cd62c3e09 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:56.269635 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2564962-f49c-4b41-85f7-654cd62c3e09 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:56.270032 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2564962-f49c-4b41-85f7-654cd62c3e09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:56.277369 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6b5199-7db3-4298-aa5a-d63542f2460b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:56.278247 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6b5199-7db3-4298-aa5a-d63542f2460b admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:56.278584 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6b5199-7db3-4298-aa5a-d63542f2460b admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:56.278920 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6b5199-7db3-4298-aa5a-d63542f2460b admin admin] PATH_INFO: `/v3/projects/9a328183e0224b7498ff5c7f3b1b0980/users/3a4dcfba9670458db6bfc880bae5e29e/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:56.300291 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee6b5199-7db3-4298-aa5a-d63542f2460b admin admin] RBAC: Authorizing `identity:create_grant(project_id=9a328183e0224b7498ff5c7f3b1b0980, user_id=3a4dcfba9670458db6bfc880bae5e29e, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:56.305706 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:56.306889 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee6b5199-7db3-4298-aa5a-d63542f2460b admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:56.320789 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-d2564962-f49c-4b41-85f7-654cd62c3e09 None None] MFA Rules not processed for user `e8c56fbde6844c45b487a81159500970`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:56.322962 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-d2564962-f49c-4b41-85f7-654cd62c3e09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:56.328434 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1021/2034] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:56:56 2022] PUT /identity/v3/projects/9a328183e0224b7498ff5c7f3b1b0980/users/3a4dcfba9670458db6bfc880bae5e29e/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:56.329487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6b5199-7db3-4298-aa5a-d63542f2460b admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:56.331502 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-ee6b5199-7db3-4298-aa5a-d63542f2460b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:56.374993 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:INSERT=6,keystone:SELECT=94 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:56.404821 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6b5199-7db3-4298-aa5a-d63542f2460b admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:56.413133 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-374bf584-4cdd-49d8-93ba-a84958e77754 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:56.414038 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-374bf584-4cdd-49d8-93ba-a84958e77754 service cinder] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:56.414369 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-374bf584-4cdd-49d8-93ba-a84958e77754 service cinder] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:56.414577 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-374bf584-4cdd-49d8-93ba-a84958e77754 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:56.419654 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1014/2035] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:56 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:56.419813 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-374bf584-4cdd-49d8-93ba-a84958e77754 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:56.419908 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2564962-f49c-4b41-85f7-654cd62c3e09 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:56.501369 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-374bf584-4cdd-49d8-93ba-a84958e77754 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:56.501369 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-374bf584-4cdd-49d8-93ba-a84958e77754 service cinder] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:56.510460 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1022/2036] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:56:56 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 181 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:56:56.511479 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-374bf584-4cdd-49d8-93ba-a84958e77754 service cinder] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:56.536402 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-374bf584-4cdd-49d8-93ba-a84958e77754 service cinder] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:56.538511 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2564962-f49c-4b41-85f7-654cd62c3e09 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:56.546481 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6788341-8936-4a26-9e0a-da08f960f0f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:56.547068 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6788341-8936-4a26-9e0a-da08f960f0f0 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:56.547343 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6788341-8936-4a26-9e0a-da08f960f0f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:56.547605 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6788341-8936-4a26-9e0a-da08f960f0f0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:56.548236 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6788341-8936-4a26-9e0a-da08f960f0f0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:56.567531 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6788341-8936-4a26-9e0a-da08f960f0f0 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:56.569189 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df5cb72-87fa-4787-a0ef-934d02f0a2e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:56.570057 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df5cb72-87fa-4787-a0ef-934d02f0a2e9 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:56.570333 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df5cb72-87fa-4787-a0ef-934d02f0a2e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:56.570594 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df5cb72-87fa-4787-a0ef-934d02f0a2e9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:56.571252 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df5cb72-87fa-4787-a0ef-934d02f0a2e9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:56.585716 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df5cb72-87fa-4787-a0ef-934d02f0a2e9 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:56.626704 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1015/2037] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:56 2022] GET /identity/v3/roles => generated 1420 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:56:56.627523 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df5cb72-87fa-4787-a0ef-934d02f0a2e9 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:56.629817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1023/2038] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:56:56 2022] POST /identity/v3/projects => generated 377 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:56:56.632175 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6788341-8936-4a26-9e0a-da08f960f0f0 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:56.636004 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-7df5cb72-87fa-4787-a0ef-934d02f0a2e9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:56.655613 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df5cb72-87fa-4787-a0ef-934d02f0a2e9 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:56.733608 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-068a1052-ca28-4f31-8eaf-d13be9b72b76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:56.734256 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068a1052-ca28-4f31-8eaf-d13be9b72b76 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:56.734533 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068a1052-ca28-4f31-8eaf-d13be9b72b76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:56.734807 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068a1052-ca28-4f31-8eaf-d13be9b72b76 admin admin] PATH_INFO: `/v3/projects/b61e2d35de03491fb4eea8e0c3789bf1` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:56.736164 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-068a1052-ca28-4f31-8eaf-d13be9b72b76 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b61e2d35de03491fb4eea8e0c3789bf1)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:56.738443 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-068a1052-ca28-4f31-8eaf-d13be9b72b76 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:56.804805 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:56.817204 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6788341-8936-4a26-9e0a-da08f960f0f0 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:56.820721 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-068a1052-ca28-4f31-8eaf-d13be9b72b76 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b61e2d35de03491fb4eea8e0c3789bf1', 'request_id': 'req-068a1052-ca28-4f31-8eaf-d13be9b72b76'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:56.837747 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-068a1052-ca28-4f31-8eaf-d13be9b72b76 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b61e2d35de03491fb4eea8e0c3789bf1', 'request_id': 'req-068a1052-ca28-4f31-8eaf-d13be9b72b76'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:56.864449 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-068a1052-ca28-4f31-8eaf-d13be9b72b76 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b61e2d35de03491fb4eea8e0c3789bf1', 'request_id': 'req-068a1052-ca28-4f31-8eaf-d13be9b72b76'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:56.884417 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-068a1052-ca28-4f31-8eaf-d13be9b72b76 admin admin] The token cache is being invalidate because project b61e2d35de03491fb4eea8e0c3789bf1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81386) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:56:56.884674 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-068a1052-ca28-4f31-8eaf-d13be9b72b76 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:56.885635 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-068a1052-ca28-4f31-8eaf-d13be9b72b76 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:56.891336 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1016/2039] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:56:56 2022] DELETE /identity/v3/projects/b61e2d35de03491fb4eea8e0c3789bf1 => generated 0 bytes in 260 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:56.891336 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-068a1052-ca28-4f31-8eaf-d13be9b72b76 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:56.891928 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-068a1052-ca28-4f31-8eaf-d13be9b72b76 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:56.987660 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad162cc-75fe-4089-8293-dff51ad274af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:56.988631 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad162cc-75fe-4089-8293-dff51ad274af admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:56.988917 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad162cc-75fe-4089-8293-dff51ad274af admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:56.989171 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad162cc-75fe-4089-8293-dff51ad274af admin admin] PATH_INFO: `/v3/projects/9a328183e0224b7498ff5c7f3b1b0980/users/3a4dcfba9670458db6bfc880bae5e29e/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:56.991952 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fad162cc-75fe-4089-8293-dff51ad274af admin admin] RBAC: Authorizing `identity:create_grant(project_id=9a328183e0224b7498ff5c7f3b1b0980, user_id=3a4dcfba9670458db6bfc880bae5e29e, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:56.994663 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fad162cc-75fe-4089-8293-dff51ad274af admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:57.014651 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1024/2040] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:56:56 2022] PUT /identity/v3/projects/9a328183e0224b7498ff5c7f3b1b0980/users/3a4dcfba9670458db6bfc880bae5e29e/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 383 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 00:56:57.015897 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be59e0a8-d521-4430-b2ac-6ee484280543 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:57.016181 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be59e0a8-d521-4430-b2ac-6ee484280543 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:57.016460 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be59e0a8-d521-4430-b2ac-6ee484280543 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:57.040041 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:57.053905 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-be59e0a8-d521-4430-b2ac-6ee484280543 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:57.055136 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-be59e0a8-d521-4430-b2ac-6ee484280543 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:57.073241 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-068a1052-ca28-4f31-8eaf-d13be9b72b76 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:57.192367 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f815419a-10ae-4e01-9f2c-860c5644c3cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:57.193221 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f815419a-10ae-4e01-9f2c-860c5644c3cd admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:57.193854 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f815419a-10ae-4e01-9f2c-860c5644c3cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:57.194190 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f815419a-10ae-4e01-9f2c-860c5644c3cd admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:57.196212 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f815419a-10ae-4e01-9f2c-860c5644c3cd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:57.197494 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1025/2041] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:57.198873 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5999b15-b58a-4dbe-b3e7-a42a7d5425f2 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:57.199174 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5999b15-b58a-4dbe-b3e7-a42a7d5425f2 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:57.199435 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5999b15-b58a-4dbe-b3e7-a42a7d5425f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:57.206546 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f815419a-10ae-4e01-9f2c-860c5644c3cd admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:57.237571 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1017/2042] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:56:56 2022] POST /identity/v3/users => generated 289 bytes in 348 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:56:57.239286 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75e1608-3f5f-49e1-8df9-61443fa6c5d8 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:57.239511 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75e1608-3f5f-49e1-8df9-61443fa6c5d8 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:57.239854 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75e1608-3f5f-49e1-8df9-61443fa6c5d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:57.244893 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-b5999b15-b58a-4dbe-b3e7-a42a7d5425f2 None None] MFA Rules not processed for user `3a4dcfba9670458db6bfc880bae5e29e`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:57.246992 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-b5999b15-b58a-4dbe-b3e7-a42a7d5425f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:57.267439 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-f75e1608-3f5f-49e1-8df9-61443fa6c5d8 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:57.268792 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-f75e1608-3f5f-49e1-8df9-61443fa6c5d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:57.282604 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1018/2043] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:57.286852 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f75e1608-3f5f-49e1-8df9-61443fa6c5d8 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:57.298427 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f75e1608-3f5f-49e1-8df9-61443fa6c5d8 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:57.304140 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16642375-401a-4c2a-9fd7-0def295bf43b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:57.304754 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16642375-401a-4c2a-9fd7-0def295bf43b admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:57.305091 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16642375-401a-4c2a-9fd7-0def295bf43b admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:57.305482 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16642375-401a-4c2a-9fd7-0def295bf43b admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:57.306358 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16642375-401a-4c2a-9fd7-0def295bf43b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:57.308315 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16642375-401a-4c2a-9fd7-0def295bf43b admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:57.324010 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1019/2044] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:57 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:56:57.324777 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16642375-401a-4c2a-9fd7-0def295bf43b admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:57.337562 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16642375-401a-4c2a-9fd7-0def295bf43b admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:57.344422 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0dd221-ce4d-49c3-b1b1-1971bef8b8a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:57.345197 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0dd221-ce4d-49c3-b1b1-1971bef8b8a6 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:57.345507 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0dd221-ce4d-49c3-b1b1-1971bef8b8a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:57.345783 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0dd221-ce4d-49c3-b1b1-1971bef8b8a6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:57.346446 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e0dd221-ce4d-49c3-b1b1-1971bef8b8a6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:56:57.346770 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e0dd221-ce4d-49c3-b1b1-1971bef8b8a6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:57.348374 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e0dd221-ce4d-49c3-b1b1-1971bef8b8a6 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:57.349024 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1026/2045] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:57 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:57.350908 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5999b15-b58a-4dbe-b3e7-a42a7d5425f2 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:57.365992 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1020/2046] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 00:56:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:56:57.375379 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0dd221-ce4d-49c3-b1b1-1971bef8b8a6 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:57.388007 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0dd221-ce4d-49c3-b1b1-1971bef8b8a6 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:57.392542 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5999b15-b58a-4dbe-b3e7-a42a7d5425f2 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:57.398856 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75845b8-4d09-4635-af57-93cc6b56c139 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:57.400548 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75845b8-4d09-4635-af57-93cc6b56c139 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:57.401095 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75845b8-4d09-4635-af57-93cc6b56c139 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:57.401811 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75845b8-4d09-4635-af57-93cc6b56c139 admin admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:57.404043 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d75845b8-4d09-4635-af57-93cc6b56c139 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:57.406391 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d75845b8-4d09-4635-af57-93cc6b56c139 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:57.421008 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-377d892b-f33d-443b-a9bb-9da17b698f7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:57.421688 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377d892b-f33d-443b-a9bb-9da17b698f7f admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:57.422048 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377d892b-f33d-443b-a9bb-9da17b698f7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:57.422328 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377d892b-f33d-443b-a9bb-9da17b698f7f admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:57.422991 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-377d892b-f33d-443b-a9bb-9da17b698f7f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:57.424609 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-377d892b-f33d-443b-a9bb-9da17b698f7f admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:57.447027 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1021/2047] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:56:57 2022] POST /identity/v3/projects => generated 369 bytes in 72 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:56:57.447521 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1027/2048] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:57 2022] GET /identity/v3/roles => generated 1420 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:56:57.448514 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75845b8-4d09-4635-af57-93cc6b56c139 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:57.449807 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-377d892b-f33d-443b-a9bb-9da17b698f7f admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:57.453349 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-d75845b8-4d09-4635-af57-93cc6b56c139 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:57.461340 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-377d892b-f33d-443b-a9bb-9da17b698f7f admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:57.472674 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=91,keystone:DELETE=4,keystone:INSERT=5 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:57.543500 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a4e5b4e-30f1-4c77-b337-a46e60b4cbf7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:57.544365 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4e5b4e-30f1-4c77-b337-a46e60b4cbf7 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:57.544715 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4e5b4e-30f1-4c77-b337-a46e60b4cbf7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:57.545079 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4e5b4e-30f1-4c77-b337-a46e60b4cbf7 admin admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:57.546615 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a4e5b4e-30f1-4c77-b337-a46e60b4cbf7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:57.546913 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75845b8-4d09-4635-af57-93cc6b56c139 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:57.551365 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b644658b-ecbe-4361-9779-9996a0566a5c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:57.552054 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b644658b-ecbe-4361-9779-9996a0566a5c service cinder] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:57.552359 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b644658b-ecbe-4361-9779-9996a0566a5c service cinder] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:57.552637 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b644658b-ecbe-4361-9779-9996a0566a5c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:57.553713 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a4e5b4e-30f1-4c77-b337-a46e60b4cbf7 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:57.591118 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1028/2049] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:56:57 2022] POST /identity/v3/users => generated 285 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:56:57.597038 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a4e5b4e-30f1-4c77-b337-a46e60b4cbf7 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:57.608319 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a4e5b4e-30f1-4c77-b337-a46e60b4cbf7 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:57.615852 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72e58bc1-e619-473d-a4b0-ded8fc0f7bed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:57.616526 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72e58bc1-e619-473d-a4b0-ded8fc0f7bed admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:57.617297 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72e58bc1-e619-473d-a4b0-ded8fc0f7bed admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:57.617297 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72e58bc1-e619-473d-a4b0-ded8fc0f7bed admin admin] PATH_INFO: `/v3/projects/1fba20633c144fe0a8572d2a2343f8c9/users/89d579de163747f2a56d0353718ecec3/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:57.620593 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:57.652382 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72e58bc1-e619-473d-a4b0-ded8fc0f7bed admin admin] RBAC: Authorizing `identity:create_grant(project_id=1fba20633c144fe0a8572d2a2343f8c9, user_id=89d579de163747f2a56d0353718ecec3, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:57.655469 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72e58bc1-e619-473d-a4b0-ded8fc0f7bed admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:57.660221 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b644658b-ecbe-4361-9779-9996a0566a5c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:57.661965 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b644658b-ecbe-4361-9779-9996a0566a5c service cinder] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:57.673179 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1029/2050] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:56:57 2022] PUT /identity/v3/projects/1fba20633c144fe0a8572d2a2343f8c9/users/89d579de163747f2a56d0353718ecec3/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 81 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:57.675794 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1022/2051] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:56:57 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 228 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:56:57.678512 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72e58bc1-e619-473d-a4b0-ded8fc0f7bed admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:57.685250 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be458bd-401e-4e66-adc0-7b2ddc85e6b0 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:57.685573 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be458bd-401e-4e66-adc0-7b2ddc85e6b0 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:57.685850 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be458bd-401e-4e66-adc0-7b2ddc85e6b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:57.749295 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-3be458bd-401e-4e66-adc0-7b2ddc85e6b0 None None] MFA Rules not processed for user `89d579de163747f2a56d0353718ecec3`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:57.750762 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-3be458bd-401e-4e66-adc0-7b2ddc85e6b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:57.800412 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72e58bc1-e619-473d-a4b0-ded8fc0f7bed admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:57.806389 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8a462e-ccf0-497f-8dfe-f3d305e00a89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:57.807016 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8a462e-ccf0-497f-8dfe-f3d305e00a89 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:57.807293 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8a462e-ccf0-497f-8dfe-f3d305e00a89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:57.807565 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8a462e-ccf0-497f-8dfe-f3d305e00a89 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:57.808214 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a8a462e-ccf0-497f-8dfe-f3d305e00a89 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:57.809911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a8a462e-ccf0-497f-8dfe-f3d305e00a89 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:57.822402 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1030/2052] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:57 2022] GET /identity/v3/roles => generated 1420 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:56:57.823440 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8a462e-ccf0-497f-8dfe-f3d305e00a89 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:57.826138 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-6a8a462e-ccf0-497f-8dfe-f3d305e00a89 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:57.840159 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8a462e-ccf0-497f-8dfe-f3d305e00a89 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:57.845366 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b84d894-1194-4368-8a00-d27acb14f097 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:57.849860 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b84d894-1194-4368-8a00-d27acb14f097 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:57.850134 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1023/2053] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:57 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:57.850249 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b84d894-1194-4368-8a00-d27acb14f097 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:57.850249 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b84d894-1194-4368-8a00-d27acb14f097 admin admin] PATH_INFO: `/v3/users/901ee7990f2c46eb913fab6b6f180e76` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:57.850396 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be458bd-401e-4e66-adc0-7b2ddc85e6b0 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:57.851402 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b84d894-1194-4368-8a00-d27acb14f097 admin admin] RBAC: Authorizing `identity:delete_user(user_id=901ee7990f2c46eb913fab6b6f180e76)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:57.853465 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b84d894-1194-4368-8a00-d27acb14f097 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:57.861242 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be458bd-401e-4e66-adc0-7b2ddc85e6b0 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:57.867612 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11300af-7930-4cd3-8bdd-27463fb3cf66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:57.868229 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11300af-7930-4cd3-8bdd-27463fb3cf66 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:57.868504 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11300af-7930-4cd3-8bdd-27463fb3cf66 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:57.868771 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11300af-7930-4cd3-8bdd-27463fb3cf66 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:57.869459 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c11300af-7930-4cd3-8bdd-27463fb3cf66 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:57.871104 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c11300af-7930-4cd3-8bdd-27463fb3cf66 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:57.884954 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1024/2054] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:57 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:56:57.885653 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11300af-7930-4cd3-8bdd-27463fb3cf66 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:57.901916 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11300af-7930-4cd3-8bdd-27463fb3cf66 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:57.906818 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29a2d93c-060b-4104-8221-27220d5713bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:57.907448 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a2d93c-060b-4104-8221-27220d5713bf admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:57.907724 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a2d93c-060b-4104-8221-27220d5713bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:57.907990 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a2d93c-060b-4104-8221-27220d5713bf admin admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:57.908661 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29a2d93c-060b-4104-8221-27220d5713bf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:57.910637 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29a2d93c-060b-4104-8221-27220d5713bf admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:57.915508 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-1b84d894-1194-4368-8a00-d27acb14f097 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '901ee7990f2c46eb913fab6b6f180e76'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:57.917102 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-1b84d894-1194-4368-8a00-d27acb14f097 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '901ee7990f2c46eb913fab6b6f180e76'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:57.927238 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-1b84d894-1194-4368-8a00-d27acb14f097 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '901ee7990f2c46eb913fab6b6f180e76'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:57.928638 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-1b84d894-1194-4368-8a00-d27acb14f097 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '901ee7990f2c46eb913fab6b6f180e76'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:57.938160 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-1b84d894-1194-4368-8a00-d27acb14f097 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '901ee7990f2c46eb913fab6b6f180e76'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:57.948886 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1025/2055] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:56:57 2022] POST /identity/v3/projects => generated 377 bytes in 64 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:56:57.953063 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29a2d93c-060b-4104-8221-27220d5713bf admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:57.954913 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-29a2d93c-060b-4104-8221-27220d5713bf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:57.972289 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1031/2056] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:56:57 2022] DELETE /identity/v3/users/901ee7990f2c46eb913fab6b6f180e76 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:57.973494 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b84d894-1194-4368-8a00-d27acb14f097 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:57.975450 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-1b84d894-1194-4368-8a00-d27acb14f097 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:57.981493 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:58.047434 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:58.052290 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29a2d93c-060b-4104-8221-27220d5713bf admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:58.059894 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d98ea9-a604-4d4f-8b43-d013f04330da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:58.060337 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d98ea9-a604-4d4f-8b43-d013f04330da admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:58.060668 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d98ea9-a604-4d4f-8b43-d013f04330da admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:58.060911 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d98ea9-a604-4d4f-8b43-d013f04330da admin admin] PATH_INFO: `/v3/projects/7750b9ec659c40a1aafc98b0107032c5/users/b1b7c24458214f77a9d447bbd22e70d1/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:58.064320 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b84d894-1194-4368-8a00-d27acb14f097 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:58.072259 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6173a8-5409-4727-8090-cb2ffd84cf8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:58.072991 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6173a8-5409-4727-8090-cb2ffd84cf8f admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:58.073392 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6173a8-5409-4727-8090-cb2ffd84cf8f admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:58.073903 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6173a8-5409-4727-8090-cb2ffd84cf8f admin admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:58.074612 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa6173a8-5409-4727-8090-cb2ffd84cf8f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:58.078096 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa6173a8-5409-4727-8090-cb2ffd84cf8f admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:58.095612 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94d98ea9-a604-4d4f-8b43-d013f04330da admin admin] RBAC: Authorizing `identity:create_grant(project_id=7750b9ec659c40a1aafc98b0107032c5, user_id=b1b7c24458214f77a9d447bbd22e70d1, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:58.098416 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94d98ea9-a604-4d4f-8b43-d013f04330da admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:58.116814 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1026/2057] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:56:57 2022] PUT /identity/v3/projects/7750b9ec659c40a1aafc98b0107032c5/users/b1b7c24458214f77a9d447bbd22e70d1/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:58.126261 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fda20a1-e714-4078-82e6-d2dbb2455c71 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:58.126261 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fda20a1-e714-4078-82e6-d2dbb2455c71 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:58.126261 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fda20a1-e714-4078-82e6-d2dbb2455c71 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:58.141795 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1032/2058] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:56:57 2022] POST /identity/v3/users => generated 295 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:56:58.141795 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248b6f5f-28e7-4344-a3e2-338916fce1e6 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:58.141795 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248b6f5f-28e7-4344-a3e2-338916fce1e6 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:58.141795 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248b6f5f-28e7-4344-a3e2-338916fce1e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:58.148054 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-8fda20a1-e714-4078-82e6-d2dbb2455c71 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:58.149741 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-8fda20a1-e714-4078-82e6-d2dbb2455c71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:58.173872 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-248b6f5f-28e7-4344-a3e2-338916fce1e6 None None] MFA Rules not processed for user `b1b7c24458214f77a9d447bbd22e70d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:58.175113 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-248b6f5f-28e7-4344-a3e2-338916fce1e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:58.238670 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1027/2059] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:58.238670 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fda20a1-e714-4078-82e6-d2dbb2455c71 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:58.256729 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fda20a1-e714-4078-82e6-d2dbb2455c71 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:58.265856 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1033/2060] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:58 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:58.267251 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-248b6f5f-28e7-4344-a3e2-338916fce1e6 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:58.267942 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae89eeb-c29a-41c4-b7b9-cba760e66e43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:58.268645 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-248b6f5f-28e7-4344-a3e2-338916fce1e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:58.270207 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae89eeb-c29a-41c4-b7b9-cba760e66e43 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:58.270207 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae89eeb-c29a-41c4-b7b9-cba760e66e43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:58.270532 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae89eeb-c29a-41c4-b7b9-cba760e66e43 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:58.273382 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aae89eeb-c29a-41c4-b7b9-cba760e66e43 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:58.277861 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aae89eeb-c29a-41c4-b7b9-cba760e66e43 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:58.293237 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1028/2061] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:58 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:56:58.294013 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae89eeb-c29a-41c4-b7b9-cba760e66e43 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:58.297958 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-aae89eeb-c29a-41c4-b7b9-cba760e66e43 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:58.348395 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-248b6f5f-28e7-4344-a3e2-338916fce1e6 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:58.353705 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e7d280-1336-4cdc-befd-f41f94570565 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:58.354324 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e7d280-1336-4cdc-befd-f41f94570565 service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:58.355179 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e7d280-1336-4cdc-befd-f41f94570565 service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:58.355447 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e7d280-1336-4cdc-befd-f41f94570565 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:58.368274 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3e7d280-1336-4cdc-befd-f41f94570565 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:58.370148 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3e7d280-1336-4cdc-befd-f41f94570565 service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:58.383171 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae89eeb-c29a-41c4-b7b9-cba760e66e43 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:58.388304 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d31a20-3ac1-4342-81fe-59a116401f9f service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4ebfc0abf1b842459a44ca9324252688', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:58.389470 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d31a20-3ac1-4342-81fe-59a116401f9f service glance] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:58.389749 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d31a20-3ac1-4342-81fe-59a116401f9f service glance] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:58.390010 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d31a20-3ac1-4342-81fe-59a116401f9f service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:58.397102 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1034/2062] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:56:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:56:58.398143 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e7d280-1336-4cdc-befd-f41f94570565 service neutron] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:58.405226 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45d31a20-3ac1-4342-81fe-59a116401f9f service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:58.406799 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45d31a20-3ac1-4342-81fe-59a116401f9f service glance] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:58.420655 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1029/2063] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:56:58 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:56:58.422274 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d31a20-3ac1-4342-81fe-59a116401f9f service glance] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:58.424549 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-45d31a20-3ac1-4342-81fe-59a116401f9f service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:58.444193 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e7d280-1336-4cdc-befd-f41f94570565 service neutron] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:58.453870 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99de9f7c-f323-47bb-9ddd-393a942a0935 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:58.453870 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99de9f7c-f323-47bb-9ddd-393a942a0935 admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:58.453870 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99de9f7c-f323-47bb-9ddd-393a942a0935 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:58.454272 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99de9f7c-f323-47bb-9ddd-393a942a0935 admin admin] PATH_INFO: `/v3/projects/bec11bdddb3e4993bbbdf5042faa8a60/users/f2b6bf09fc3846e6ad42b0e406c45e7f/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:58.489401 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99de9f7c-f323-47bb-9ddd-393a942a0935 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bec11bdddb3e4993bbbdf5042faa8a60, user_id=f2b6bf09fc3846e6ad42b0e406c45e7f, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:58.494297 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99de9f7c-f323-47bb-9ddd-393a942a0935 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:58.497780 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:58.527814 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1035/2064] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:56:58 2022] PUT /identity/v3/projects/bec11bdddb3e4993bbbdf5042faa8a60/users/f2b6bf09fc3846e6ad42b0e406c45e7f/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:58.536535 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99de9f7c-f323-47bb-9ddd-393a942a0935 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:58.579492 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d31a20-3ac1-4342-81fe-59a116401f9f service glance] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:58.703632 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c5cbe4-684d-4365-afe2-71144b3b59ed service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:58.704579 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c5cbe4-684d-4365-afe2-71144b3b59ed service cinder] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:58.705371 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c5cbe4-684d-4365-afe2-71144b3b59ed service cinder] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:58.705647 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c5cbe4-684d-4365-afe2-71144b3b59ed service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:58.721341 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-d2c5cbe4-684d-4365-afe2-71144b3b59ed service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:58.765313 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:58.799486 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99de9f7c-f323-47bb-9ddd-393a942a0935 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:58.810824 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f49769c-55f7-4863-96c2-c0bd5920f2de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:58.811798 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f49769c-55f7-4863-96c2-c0bd5920f2de admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:58.811798 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f49769c-55f7-4863-96c2-c0bd5920f2de admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:58.811987 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f49769c-55f7-4863-96c2-c0bd5920f2de admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:58.812605 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f49769c-55f7-4863-96c2-c0bd5920f2de admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:58.817822 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f49769c-55f7-4863-96c2-c0bd5920f2de admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:58.842366 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1036/2065] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:56:58 2022] GET /identity/v3/roles => generated 1420 bytes in 307 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:56:58.844026 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f49769c-55f7-4863-96c2-c0bd5920f2de admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:58.867883 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f49769c-55f7-4863-96c2-c0bd5920f2de admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:58.879032 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c5cbe4-684d-4365-afe2-71144b3b59ed service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:58.881905 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c5cbe4-684d-4365-afe2-71144b3b59ed service cinder] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:58.885086 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0c7b00-9af2-4d90-b7ba-463e6b26072f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:58.885772 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0c7b00-9af2-4d90-b7ba-463e6b26072f admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:58.886123 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0c7b00-9af2-4d90-b7ba-463e6b26072f admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:58.886608 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0c7b00-9af2-4d90-b7ba-463e6b26072f admin admin] PATH_INFO: `/v3/projects/bec11bdddb3e4993bbbdf5042faa8a60/users/f2b6bf09fc3846e6ad42b0e406c45e7f/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:58.890880 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb0c7b00-9af2-4d90-b7ba-463e6b26072f admin admin] RBAC: Authorizing `identity:create_grant(project_id=bec11bdddb3e4993bbbdf5042faa8a60, user_id=f2b6bf09fc3846e6ad42b0e406c45e7f, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:58.895927 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1030/2066] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:56:58 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 474 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:56:58.900918 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb0c7b00-9af2-4d90-b7ba-463e6b26072f admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:58.918521 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1037/2067] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:56:58 2022] PUT /identity/v3/projects/bec11bdddb3e4993bbbdf5042faa8a60/users/f2b6bf09fc3846e6ad42b0e406c45e7f/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 75 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:56:58.931419 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29eb56b0-62eb-4e03-b66e-74153a46aa9a None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:58.931699 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29eb56b0-62eb-4e03-b66e-74153a46aa9a None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:58.931950 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29eb56b0-62eb-4e03-b66e-74153a46aa9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:58.959079 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0c7b00-9af2-4d90-b7ba-463e6b26072f admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:58.960550 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-eb0c7b00-9af2-4d90-b7ba-463e6b26072f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:59.012885 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-29eb56b0-62eb-4e03-b66e-74153a46aa9a None None] MFA Rules not processed for user `f2b6bf09fc3846e6ad42b0e406c45e7f`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:56:59.016284 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-29eb56b0-62eb-4e03-b66e-74153a46aa9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:59.168170 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0c7b00-9af2-4d90-b7ba-463e6b26072f admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:59.188321 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc37f9a1-b0fa-4146-aa6d-9a586bd57e56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:59.188974 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc37f9a1-b0fa-4146-aa6d-9a586bd57e56 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:59.189285 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc37f9a1-b0fa-4146-aa6d-9a586bd57e56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:59.189628 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc37f9a1-b0fa-4146-aa6d-9a586bd57e56 admin admin] PATH_INFO: `/v3/projects/7b7dff005bcf45ce9fb79fdc2b8eab52` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:59.192790 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc37f9a1-b0fa-4146-aa6d-9a586bd57e56 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7b7dff005bcf45ce9fb79fdc2b8eab52)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:59.195212 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc37f9a1-b0fa-4146-aa6d-9a586bd57e56 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:59.232424 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1031/2068] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:56:58 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 303 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:56:59.250960 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29eb56b0-62eb-4e03-b66e-74153a46aa9a None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:59.317741 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:59.341099 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-cc37f9a1-b0fa-4146-aa6d-9a586bd57e56 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7b7dff005bcf45ce9fb79fdc2b8eab52', 'request_id': 'req-cc37f9a1-b0fa-4146-aa6d-9a586bd57e56'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:59.360530 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-cc37f9a1-b0fa-4146-aa6d-9a586bd57e56 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7b7dff005bcf45ce9fb79fdc2b8eab52', 'request_id': 'req-cc37f9a1-b0fa-4146-aa6d-9a586bd57e56'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:59.379874 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-cc37f9a1-b0fa-4146-aa6d-9a586bd57e56 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7b7dff005bcf45ce9fb79fdc2b8eab52', 'request_id': 'req-cc37f9a1-b0fa-4146-aa6d-9a586bd57e56'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:59.409117 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29eb56b0-62eb-4e03-b66e-74153a46aa9a None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:59.419405 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-cc37f9a1-b0fa-4146-aa6d-9a586bd57e56 admin admin] The token cache is being invalidate because project 7b7dff005bcf45ce9fb79fdc2b8eab52 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:56:59.419704 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-cc37f9a1-b0fa-4146-aa6d-9a586bd57e56 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:59.423855 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-cc37f9a1-b0fa-4146-aa6d-9a586bd57e56 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:59.429442 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1038/2069] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:56:58 2022] DELETE /identity/v3/projects/7b7dff005bcf45ce9fb79fdc2b8eab52 => generated 0 bytes in 468 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 00:56:59.430397 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc37f9a1-b0fa-4146-aa6d-9a586bd57e56 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:56:59.431822 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-cc37f9a1-b0fa-4146-aa6d-9a586bd57e56 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:59.592743 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc37f9a1-b0fa-4146-aa6d-9a586bd57e56 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:56:59.611148 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eba0638-d7ab-412f-9341-f9b710e9130f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:59.611925 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eba0638-d7ab-412f-9341-f9b710e9130f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:59.612280 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eba0638-d7ab-412f-9341-f9b710e9130f admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:59.612622 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eba0638-d7ab-412f-9341-f9b710e9130f admin admin] PATH_INFO: `/v3/users/3a4dcfba9670458db6bfc880bae5e29e` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:59.614194 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eba0638-d7ab-412f-9341-f9b710e9130f admin admin] RBAC: Authorizing `identity:delete_user(user_id=3a4dcfba9670458db6bfc880bae5e29e)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:59.616948 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eba0638-d7ab-412f-9341-f9b710e9130f admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:59.626719 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c5231c-febc-40b6-b339-350529063dbf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:56:59.627804 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c5231c-febc-40b6-b339-350529063dbf service cinder] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:56:59.628285 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c5231c-febc-40b6-b339-350529063dbf service cinder] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:56:59.628767 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c5231c-febc-40b6-b339-350529063dbf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:56:59.632769 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-a6c5231c-febc-40b6-b339-350529063dbf service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:56:59.646751 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:56:59.730546 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c5231c-febc-40b6-b339-350529063dbf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:56:59.733362 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c5231c-febc-40b6-b339-350529063dbf service cinder] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:56:59.739130 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-1eba0638-d7ab-412f-9341-f9b710e9130f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3a4dcfba9670458db6bfc880bae5e29e'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:59.744914 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-1eba0638-d7ab-412f-9341-f9b710e9130f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3a4dcfba9670458db6bfc880bae5e29e'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:59.752157 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1032/2070] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:56:59 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 502 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:56:59.765760 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-1eba0638-d7ab-412f-9341-f9b710e9130f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3a4dcfba9670458db6bfc880bae5e29e'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:59.768330 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-1eba0638-d7ab-412f-9341-f9b710e9130f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3a4dcfba9670458db6bfc880bae5e29e'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:59.791846 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-1eba0638-d7ab-412f-9341-f9b710e9130f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3a4dcfba9670458db6bfc880bae5e29e'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:56:59.825690 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1039/2071] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:56:59 2022] DELETE /identity/v3/users/3a4dcfba9670458db6bfc880bae5e29e => generated 0 bytes in 396 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:57:00.317963 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c5231c-febc-40b6-b339-350529063dbf service cinder] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:00.335348 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c5231c-febc-40b6-b339-350529063dbf service cinder] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:00.417630 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeb45b99-433d-425d-9d26-41fb944c5096 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:00.418272 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb45b99-433d-425d-9d26-41fb944c5096 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:00.418684 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb45b99-433d-425d-9d26-41fb944c5096 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:00.418850 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb45b99-433d-425d-9d26-41fb944c5096 admin admin] PATH_INFO: `/v3/projects/9a328183e0224b7498ff5c7f3b1b0980` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:00.420024 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeb45b99-433d-425d-9d26-41fb944c5096 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9a328183e0224b7498ff5c7f3b1b0980)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:00.422021 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeb45b99-433d-425d-9d26-41fb944c5096 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:00.466971 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:57:00.482959 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-eeb45b99-433d-425d-9d26-41fb944c5096 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9a328183e0224b7498ff5c7f3b1b0980', 'request_id': 'req-eeb45b99-433d-425d-9d26-41fb944c5096'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:00.494402 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-eeb45b99-433d-425d-9d26-41fb944c5096 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9a328183e0224b7498ff5c7f3b1b0980', 'request_id': 'req-eeb45b99-433d-425d-9d26-41fb944c5096'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:00.503317 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-eeb45b99-433d-425d-9d26-41fb944c5096 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9a328183e0224b7498ff5c7f3b1b0980', 'request_id': 'req-eeb45b99-433d-425d-9d26-41fb944c5096'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:00.515915 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-eeb45b99-433d-425d-9d26-41fb944c5096 admin admin] The token cache is being invalidate because project 9a328183e0224b7498ff5c7f3b1b0980 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81386) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:57:00.516225 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-eeb45b99-433d-425d-9d26-41fb944c5096 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:00.516857 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-eeb45b99-433d-425d-9d26-41fb944c5096 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:00.518777 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1033/2072] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:57:00 2022] DELETE /identity/v3/projects/9a328183e0224b7498ff5c7f3b1b0980 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:57:00.525229 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca713e2c-562c-4930-ae5d-d384c79be90e None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:00.525810 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca713e2c-562c-4930-ae5d-d384c79be90e None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:00.525810 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca713e2c-562c-4930-ae5d-d384c79be90e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:00.547231 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-ca713e2c-562c-4930-ae5d-d384c79be90e None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:57:00.548497 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-ca713e2c-562c-4930-ae5d-d384c79be90e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:00.619140 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1040/2073] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:57:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:57:00.628550 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea0bdef4-935d-4f33-badf-9b8de003fe8b None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:00.628830 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea0bdef4-935d-4f33-badf-9b8de003fe8b None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:00.629098 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea0bdef4-935d-4f33-badf-9b8de003fe8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:00.651976 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-ea0bdef4-935d-4f33-badf-9b8de003fe8b None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:57:00.653386 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-ea0bdef4-935d-4f33-badf-9b8de003fe8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:00.665409 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1034/2074] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:57:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:57:00.667529 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca713e2c-562c-4930-ae5d-d384c79be90e None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:00.678075 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca713e2c-562c-4930-ae5d-d384c79be90e None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:00.684293 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c070d4d-d09a-41b3-bc3a-3f627f3fc765 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:00.685186 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c070d4d-d09a-41b3-bc3a-3f627f3fc765 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:00.685501 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c070d4d-d09a-41b3-bc3a-3f627f3fc765 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:00.685767 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c070d4d-d09a-41b3-bc3a-3f627f3fc765 admin admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:00.687139 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c070d4d-d09a-41b3-bc3a-3f627f3fc765 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:57:00.687448 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c070d4d-d09a-41b3-bc3a-3f627f3fc765 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:00.689716 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c070d4d-d09a-41b3-bc3a-3f627f3fc765 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:00.705300 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1041/2075] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 00:57:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:57:00.709392 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea0bdef4-935d-4f33-badf-9b8de003fe8b None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:00.719822 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea0bdef4-935d-4f33-badf-9b8de003fe8b None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:00.728146 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7ebaf1-fce9-41fd-96b8-de4c513780d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:00.729187 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7ebaf1-fce9-41fd-96b8-de4c513780d5 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:00.729548 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7ebaf1-fce9-41fd-96b8-de4c513780d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:00.729749 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7ebaf1-fce9-41fd-96b8-de4c513780d5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:00.730393 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f7ebaf1-fce9-41fd-96b8-de4c513780d5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:00.732322 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f7ebaf1-fce9-41fd-96b8-de4c513780d5 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:00.805785 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1035/2076] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:57:00 2022] POST /identity/v3/projects => generated 399 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:57:00.810584 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c070d4d-d09a-41b3-bc3a-3f627f3fc765 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:00.824870 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c070d4d-d09a-41b3-bc3a-3f627f3fc765 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:00.950066 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbdbd5b5-9b00-430a-aee7-b79bb527ac58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:00.950886 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbdbd5b5-9b00-430a-aee7-b79bb527ac58 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:00.951317 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbdbd5b5-9b00-430a-aee7-b79bb527ac58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:00.951827 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbdbd5b5-9b00-430a-aee7-b79bb527ac58 admin admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:00.952549 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbdbd5b5-9b00-430a-aee7-b79bb527ac58 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:00.955449 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbdbd5b5-9b00-430a-aee7-b79bb527ac58 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:00.988938 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:57:00.997116 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1042/2077] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:57:00 2022] POST /identity/v3/users => generated 300 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:57:00.999969 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7ebaf1-fce9-41fd-96b8-de4c513780d5 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:01.012775 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7ebaf1-fce9-41fd-96b8-de4c513780d5 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:01.018328 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e76867-bdb9-44ce-bbd9-151e1c20415b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:01.018996 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e76867-bdb9-44ce-bbd9-151e1c20415b admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:01.019269 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e76867-bdb9-44ce-bbd9-151e1c20415b admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:01.019537 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e76867-bdb9-44ce-bbd9-151e1c20415b admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:01.020230 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95e76867-bdb9-44ce-bbd9-151e1c20415b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:01.022534 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95e76867-bdb9-44ce-bbd9-151e1c20415b admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:01.038144 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1036/2078] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:57:00 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:57:01.042437 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbdbd5b5-9b00-430a-aee7-b79bb527ac58 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:01.055791 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbdbd5b5-9b00-430a-aee7-b79bb527ac58 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:01.072802 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbee5b43-4377-4806-bf84-471b46074620 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:01.073809 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbee5b43-4377-4806-bf84-471b46074620 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:01.074126 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbee5b43-4377-4806-bf84-471b46074620 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:01.074431 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbee5b43-4377-4806-bf84-471b46074620 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:01.075254 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbee5b43-4377-4806-bf84-471b46074620 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:01.077040 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbee5b43-4377-4806-bf84-471b46074620 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:01.094370 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1043/2079] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:57:01 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:57:01.095069 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e76867-bdb9-44ce-bbd9-151e1c20415b admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:01.115397 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e76867-bdb9-44ce-bbd9-151e1c20415b admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:01.133673 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b9dace7-177d-4e47-89d1-5035c604f060 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:01.134344 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b9dace7-177d-4e47-89d1-5035c604f060 admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:01.134764 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b9dace7-177d-4e47-89d1-5035c604f060 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:01.135111 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b9dace7-177d-4e47-89d1-5035c604f060 admin admin] PATH_INFO: `/v3/projects/4bccc5e412f848b3957c110e731c309d/users/2febc356e51b410583c403c09702f2f3/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:01.153790 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b9dace7-177d-4e47-89d1-5035c604f060 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4bccc5e412f848b3957c110e731c309d, user_id=2febc356e51b410583c403c09702f2f3, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:01.156415 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b9dace7-177d-4e47-89d1-5035c604f060 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:01.171116 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1037/2080] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:57:01 2022] PUT /identity/v3/projects/4bccc5e412f848b3957c110e731c309d/users/2febc356e51b410583c403c09702f2f3/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 77 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 00:57:01.183261 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa68cb4e-47bc-45c2-88dd-01396b590b19 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:01.183538 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa68cb4e-47bc-45c2-88dd-01396b590b19 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:01.183832 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa68cb4e-47bc-45c2-88dd-01396b590b19 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:01.217115 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-aa68cb4e-47bc-45c2-88dd-01396b590b19 None None] MFA Rules not processed for user `2febc356e51b410583c403c09702f2f3`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:57:01.218377 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-aa68cb4e-47bc-45c2-88dd-01396b590b19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:01.284732 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1044/2081] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:57:01 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:57:01.290086 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b9dace7-177d-4e47-89d1-5035c604f060 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:01.352838 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b9dace7-177d-4e47-89d1-5035c604f060 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:01.358031 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b83914-0f81-4f90-9243-8cf9d5911a95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:01.358643 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b83914-0f81-4f90-9243-8cf9d5911a95 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:01.358920 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b83914-0f81-4f90-9243-8cf9d5911a95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:01.359196 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b83914-0f81-4f90-9243-8cf9d5911a95 admin admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:01.359901 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b83914-0f81-4f90-9243-8cf9d5911a95 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:01.361777 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b83914-0f81-4f90-9243-8cf9d5911a95 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:01.394658 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1038/2082] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:57:01 2022] POST /identity/v3/projects => generated 401 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:57:01.395784 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa68cb4e-47bc-45c2-88dd-01396b590b19 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:01.409223 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa68cb4e-47bc-45c2-88dd-01396b590b19 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:01.476632 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f4c4e9-b58c-46c6-b705-ad70c4ee0270 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:01.477240 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f4c4e9-b58c-46c6-b705-ad70c4ee0270 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:01.477571 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f4c4e9-b58c-46c6-b705-ad70c4ee0270 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:01.479246 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f4c4e9-b58c-46c6-b705-ad70c4ee0270 admin admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:01.479246 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f4c4e9-b58c-46c6-b705-ad70c4ee0270 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:01.480688 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f4c4e9-b58c-46c6-b705-ad70c4ee0270 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:01.509844 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1045/2083] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:57:01 2022] POST /identity/v3/users => generated 307 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:57:01.511028 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b83914-0f81-4f90-9243-8cf9d5911a95 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:01.522001 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b83914-0f81-4f90-9243-8cf9d5911a95 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:01.526799 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533b19f8-85b0-4513-90c6-ec637642172b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:01.527398 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533b19f8-85b0-4513-90c6-ec637642172b admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:01.527935 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533b19f8-85b0-4513-90c6-ec637642172b admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:01.527935 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533b19f8-85b0-4513-90c6-ec637642172b admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:01.528558 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-533b19f8-85b0-4513-90c6-ec637642172b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:01.530183 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-533b19f8-85b0-4513-90c6-ec637642172b admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:01.544302 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1039/2084] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:57:01 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:57:01.545931 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f4c4e9-b58c-46c6-b705-ad70c4ee0270 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:01.557571 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f4c4e9-b58c-46c6-b705-ad70c4ee0270 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:01.562266 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d2bcc4a-0319-44bd-88f4-a10a629e1180 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:01.562914 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d2bcc4a-0319-44bd-88f4-a10a629e1180 admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:01.563187 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d2bcc4a-0319-44bd-88f4-a10a629e1180 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:01.563451 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d2bcc4a-0319-44bd-88f4-a10a629e1180 admin admin] PATH_INFO: `/v3/projects/d410822d4a0e49029e0d62cab39bbd45/users/797ea6f2ee7b41098addbabc4be4e487/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:01.581345 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d2bcc4a-0319-44bd-88f4-a10a629e1180 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d410822d4a0e49029e0d62cab39bbd45, user_id=797ea6f2ee7b41098addbabc4be4e487, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:01.584825 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d2bcc4a-0319-44bd-88f4-a10a629e1180 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:01.598795 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1046/2085] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:57:01 2022] PUT /identity/v3/projects/d410822d4a0e49029e0d62cab39bbd45/users/797ea6f2ee7b41098addbabc4be4e487/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 00:57:01.602817 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533b19f8-85b0-4513-90c6-ec637642172b admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:01.613521 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533b19f8-85b0-4513-90c6-ec637642172b admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:01.621510 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d2bcc4a-0319-44bd-88f4-a10a629e1180 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:01.623220 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-4d2bcc4a-0319-44bd-88f4-a10a629e1180 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:01.664601 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=92,keystone:DELETE=4,keystone:INSERT=4 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:57:01.685485 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:57:01.692890 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e158dbcd-dff3-491d-bab6-b3cbacb44ad7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:01.693603 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e158dbcd-dff3-491d-bab6-b3cbacb44ad7 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:01.693879 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e158dbcd-dff3-491d-bab6-b3cbacb44ad7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:01.694149 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e158dbcd-dff3-491d-bab6-b3cbacb44ad7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:01.694789 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e158dbcd-dff3-491d-bab6-b3cbacb44ad7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:01.696399 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e158dbcd-dff3-491d-bab6-b3cbacb44ad7 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:01.697871 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d2bcc4a-0319-44bd-88f4-a10a629e1180 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:01.702499 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-921ca641-4d88-4708-b193-d784ca4a69bd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:01.703293 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921ca641-4d88-4708-b193-d784ca4a69bd service cinder] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:01.703572 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921ca641-4d88-4708-b193-d784ca4a69bd service cinder] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:01.703976 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921ca641-4d88-4708-b193-d784ca4a69bd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:01.706948 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1040/2086] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:57:01 2022] GET /identity/v3/roles => generated 1420 bytes in 105 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 00:57:01.710484 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-921ca641-4d88-4708-b193-d784ca4a69bd service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:01.712864 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e158dbcd-dff3-491d-bab6-b3cbacb44ad7 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:01.725846 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e158dbcd-dff3-491d-bab6-b3cbacb44ad7 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:01.731135 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f8e7af-a924-4572-82b6-a0471149e3dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:01.731843 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f8e7af-a924-4572-82b6-a0471149e3dc admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:01.732117 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f8e7af-a924-4572-82b6-a0471149e3dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:01.732382 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f8e7af-a924-4572-82b6-a0471149e3dc admin admin] PATH_INFO: `/v3/projects/d410822d4a0e49029e0d62cab39bbd45/users/797ea6f2ee7b41098addbabc4be4e487/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:01.734342 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f8e7af-a924-4572-82b6-a0471149e3dc admin admin] RBAC: Authorizing `identity:create_grant(project_id=d410822d4a0e49029e0d62cab39bbd45, user_id=797ea6f2ee7b41098addbabc4be4e487, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:01.737077 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f8e7af-a924-4572-82b6-a0471149e3dc admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:01.749015 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1041/2087] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:57:01 2022] PUT /identity/v3/projects/d410822d4a0e49029e0d62cab39bbd45/users/797ea6f2ee7b41098addbabc4be4e487/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 00:57:01.759278 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e310efe-23e0-41dc-b054-cd1f7335fd7f None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:01.759560 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e310efe-23e0-41dc-b054-cd1f7335fd7f None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:01.759830 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e310efe-23e0-41dc-b054-cd1f7335fd7f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:01.778334 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-921ca641-4d88-4708-b193-d784ca4a69bd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:01.779724 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-921ca641-4d88-4708-b193-d784ca4a69bd service cinder] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:01.788775 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-6e310efe-23e0-41dc-b054-cd1f7335fd7f None None] MFA Rules not processed for user `797ea6f2ee7b41098addbabc4be4e487`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:57:01.789382 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1047/2088] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:57:01 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 00:57:01.790044 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-6e310efe-23e0-41dc-b054-cd1f7335fd7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:01.862096 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1042/2089] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:57:01 2022] POST /identity/v3/auth/tokens => generated 3345 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 00:57:01.883996 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-921ca641-4d88-4708-b193-d784ca4a69bd service cinder] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:01.885498 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-921ca641-4d88-4708-b193-d784ca4a69bd service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:01.897847 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-921ca641-4d88-4708-b193-d784ca4a69bd service cinder] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:01.962356 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6ddd33a-2936-47c8-b944-98c9252aef02 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:01.962998 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ddd33a-2936-47c8-b944-98c9252aef02 service cinder] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:01.963263 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ddd33a-2936-47c8-b944-98c9252aef02 service cinder] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:01.963531 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ddd33a-2936-47c8-b944-98c9252aef02 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:01.973893 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6ddd33a-2936-47c8-b944-98c9252aef02 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:01.975751 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6ddd33a-2936-47c8-b944-98c9252aef02 service cinder] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:01.990614 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1048/2090] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:57:01 2022] GET /identity/v3/auth/tokens => generated 3345 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:57:02.151528 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e310efe-23e0-41dc-b054-cd1f7335fd7f None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:02.166577 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e310efe-23e0-41dc-b054-cd1f7335fd7f None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:02.174280 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae57037-4c13-409a-b332-fd066d9d39dc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:02.174882 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae57037-4c13-409a-b332-fd066d9d39dc service cinder] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:02.175157 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae57037-4c13-409a-b332-fd066d9d39dc service cinder] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:02.175425 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae57037-4c13-409a-b332-fd066d9d39dc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:02.254716 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae57037-4c13-409a-b332-fd066d9d39dc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:02.256834 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae57037-4c13-409a-b332-fd066d9d39dc service cinder] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:02.266879 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1043/2091] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:57:02 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:57:02.578087 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6ddd33a-2936-47c8-b944-98c9252aef02 service cinder] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:02.587586 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6ddd33a-2936-47c8-b944-98c9252aef02 service cinder] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:02.591985 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5055aa8-b1b7-446d-945f-3d573d7dbac6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:02.592748 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5055aa8-b1b7-446d-945f-3d573d7dbac6 service cinder] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:02.593039 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5055aa8-b1b7-446d-945f-3d573d7dbac6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:02.593339 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5055aa8-b1b7-446d-945f-3d573d7dbac6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:02.595522 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-d5055aa8-b1b7-446d-945f-3d573d7dbac6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:02.695400 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5055aa8-b1b7-446d-945f-3d573d7dbac6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:02.696979 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5055aa8-b1b7-446d-945f-3d573d7dbac6 service cinder] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:02.707689 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1049/2092] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:57:02 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:57:06.499989 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae57037-4c13-409a-b332-fd066d9d39dc service cinder] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:06.545661 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:57:06.565229 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae57037-4c13-409a-b332-fd066d9d39dc service cinder] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:06.570057 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c05cd0fa-d6a4-4cb5-a9a9-fd926809ab4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:06.570680 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05cd0fa-d6a4-4cb5-a9a9-fd926809ab4a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:06.570954 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05cd0fa-d6a4-4cb5-a9a9-fd926809ab4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:06.571220 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05cd0fa-d6a4-4cb5-a9a9-fd926809ab4a admin admin] PATH_INFO: `/v3/users/2febc356e51b410583c403c09702f2f3` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:06.572324 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c05cd0fa-d6a4-4cb5-a9a9-fd926809ab4a admin admin] RBAC: Authorizing `identity:delete_user(user_id=2febc356e51b410583c403c09702f2f3)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:06.574348 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c05cd0fa-d6a4-4cb5-a9a9-fd926809ab4a admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:06.635095 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-c05cd0fa-d6a4-4cb5-a9a9-fd926809ab4a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2febc356e51b410583c403c09702f2f3'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:06.641140 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-c05cd0fa-d6a4-4cb5-a9a9-fd926809ab4a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2febc356e51b410583c403c09702f2f3'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:06.641931 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-c05cd0fa-d6a4-4cb5-a9a9-fd926809ab4a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2febc356e51b410583c403c09702f2f3'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:06.652033 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-c05cd0fa-d6a4-4cb5-a9a9-fd926809ab4a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2febc356e51b410583c403c09702f2f3'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:06.652781 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-c05cd0fa-d6a4-4cb5-a9a9-fd926809ab4a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2febc356e51b410583c403c09702f2f3'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:06.672435 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1044/2093] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:57:06 2022] DELETE /identity/v3/users/2febc356e51b410583c403c09702f2f3 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:57:06.676879 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f43673a-00e3-42e7-b9dd-480c6dec0179 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:06.678076 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f43673a-00e3-42e7-b9dd-480c6dec0179 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:06.678337 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f43673a-00e3-42e7-b9dd-480c6dec0179 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:06.698215 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-1f43673a-00e3-42e7-b9dd-480c6dec0179 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:57:06.699412 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-1f43673a-00e3-42e7-b9dd-480c6dec0179 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:06.733507 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:57:06.769437 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1050/2094] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:57:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:57:06.780039 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c05cd0fa-d6a4-4cb5-a9a9-fd926809ab4a admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:06.782766 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-c05cd0fa-d6a4-4cb5-a9a9-fd926809ab4a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:06.860883 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c05cd0fa-d6a4-4cb5-a9a9-fd926809ab4a admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:06.871702 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db6145e3-90b6-4366-af02-7bcb7decb860 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:06.872485 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6145e3-90b6-4366-af02-7bcb7decb860 service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:06.872959 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6145e3-90b6-4366-af02-7bcb7decb860 service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:06.873342 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6145e3-90b6-4366-af02-7bcb7decb860 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:06.887404 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db6145e3-90b6-4366-af02-7bcb7decb860 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:06.889554 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db6145e3-90b6-4366-af02-7bcb7decb860 service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:06.914803 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1045/2095] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:57:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:57:06.967057 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f43673a-00e3-42e7-b9dd-480c6dec0179 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:06.969465 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-1f43673a-00e3-42e7-b9dd-480c6dec0179 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:07.022401 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f43673a-00e3-42e7-b9dd-480c6dec0179 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:07.033687 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1176bc30-21c5-47cc-b1a7-03c7ba41dfc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:07.035011 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1176bc30-21c5-47cc-b1a7-03c7ba41dfc6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:07.035500 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1176bc30-21c5-47cc-b1a7-03c7ba41dfc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:07.035966 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1176bc30-21c5-47cc-b1a7-03c7ba41dfc6 admin admin] PATH_INFO: `/v3/users/e8c56fbde6844c45b487a81159500970` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:07.038947 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1176bc30-21c5-47cc-b1a7-03c7ba41dfc6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e8c56fbde6844c45b487a81159500970)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:07.040927 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1176bc30-21c5-47cc-b1a7-03c7ba41dfc6 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:07.095819 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-1176bc30-21c5-47cc-b1a7-03c7ba41dfc6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e8c56fbde6844c45b487a81159500970'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:07.096574 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-1176bc30-21c5-47cc-b1a7-03c7ba41dfc6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e8c56fbde6844c45b487a81159500970'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:07.106124 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-1176bc30-21c5-47cc-b1a7-03c7ba41dfc6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e8c56fbde6844c45b487a81159500970'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:07.106764 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-1176bc30-21c5-47cc-b1a7-03c7ba41dfc6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e8c56fbde6844c45b487a81159500970'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:07.116589 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-1176bc30-21c5-47cc-b1a7-03c7ba41dfc6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e8c56fbde6844c45b487a81159500970'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:07.137529 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1051/2096] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:57:06 2022] DELETE /identity/v3/users/e8c56fbde6844c45b487a81159500970 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:57:07.141947 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1462908b-ce10-4f61-983d-4987e0a4152f None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:07.142229 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1462908b-ce10-4f61-983d-4987e0a4152f None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:07.142518 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1462908b-ce10-4f61-983d-4987e0a4152f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:07.163046 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-1462908b-ce10-4f61-983d-4987e0a4152f None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:57:07.164333 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-1462908b-ce10-4f61-983d-4987e0a4152f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:07.226703 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1046/2097] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:57:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:57:07.229568 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1176bc30-21c5-47cc-b1a7-03c7ba41dfc6 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:07.231067 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-1176bc30-21c5-47cc-b1a7-03c7ba41dfc6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:07.235923 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1462908b-ce10-4f61-983d-4987e0a4152f None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:07.237345 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-1462908b-ce10-4f61-983d-4987e0a4152f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:07.246387 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:57:07.253893 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1176bc30-21c5-47cc-b1a7-03c7ba41dfc6 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:07.261547 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ec50bc-ee91-4d1c-9308-8317c6e5c065 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:07.262163 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ec50bc-ee91-4d1c-9308-8317c6e5c065 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:07.262443 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ec50bc-ee91-4d1c-9308-8317c6e5c065 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:07.262714 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ec50bc-ee91-4d1c-9308-8317c6e5c065 admin admin] PATH_INFO: `/v3/projects/4bccc5e412f848b3957c110e731c309d` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:07.263884 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0ec50bc-ee91-4d1c-9308-8317c6e5c065 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4bccc5e412f848b3957c110e731c309d)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:07.266595 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0ec50bc-ee91-4d1c-9308-8317c6e5c065 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:07.306769 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1462908b-ce10-4f61-983d-4987e0a4152f None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:07.315579 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-b0ec50bc-ee91-4d1c-9308-8317c6e5c065 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4bccc5e412f848b3957c110e731c309d', 'request_id': 'req-b0ec50bc-ee91-4d1c-9308-8317c6e5c065'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:07.326171 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-b0ec50bc-ee91-4d1c-9308-8317c6e5c065 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4bccc5e412f848b3957c110e731c309d', 'request_id': 'req-b0ec50bc-ee91-4d1c-9308-8317c6e5c065'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:07.332308 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-b0ec50bc-ee91-4d1c-9308-8317c6e5c065 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4bccc5e412f848b3957c110e731c309d', 'request_id': 'req-b0ec50bc-ee91-4d1c-9308-8317c6e5c065'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:07.345096 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-b0ec50bc-ee91-4d1c-9308-8317c6e5c065 admin admin] The token cache is being invalidate because project 4bccc5e412f848b3957c110e731c309d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:57:07.345315 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-b0ec50bc-ee91-4d1c-9308-8317c6e5c065 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:07.345964 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-b0ec50bc-ee91-4d1c-9308-8317c6e5c065 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:07.347491 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1052/2098] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:57:07 2022] DELETE /identity/v3/projects/4bccc5e412f848b3957c110e731c309d => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:57:07.351786 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ec50bc-ee91-4d1c-9308-8317c6e5c065 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:07.353431 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-b0ec50bc-ee91-4d1c-9308-8317c6e5c065 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:07.357158 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22aa4257-5b75-47c1-86a2-621e9d013998 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:07.357793 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22aa4257-5b75-47c1-86a2-621e9d013998 service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:07.358075 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22aa4257-5b75-47c1-86a2-621e9d013998 service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:07.358336 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22aa4257-5b75-47c1-86a2-621e9d013998 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:07.361873 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-22aa4257-5b75-47c1-86a2-621e9d013998 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:07.394153 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:57:07.430199 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ec50bc-ee91-4d1c-9308-8317c6e5c065 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:07.434981 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08cab540-038f-47eb-9c09-53c740f4cb56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:07.435459 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22aa4257-5b75-47c1-86a2-621e9d013998 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:07.435774 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08cab540-038f-47eb-9c09-53c740f4cb56 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:07.435878 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08cab540-038f-47eb-9c09-53c740f4cb56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:07.436170 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08cab540-038f-47eb-9c09-53c740f4cb56 admin admin] PATH_INFO: `/v3/users/797ea6f2ee7b41098addbabc4be4e487` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:07.436900 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22aa4257-5b75-47c1-86a2-621e9d013998 service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:07.437328 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08cab540-038f-47eb-9c09-53c740f4cb56 admin admin] RBAC: Authorizing `identity:delete_user(user_id=797ea6f2ee7b41098addbabc4be4e487)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:07.439304 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08cab540-038f-47eb-9c09-53c740f4cb56 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:07.458859 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1047/2099] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:57:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 223 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:57:07.508987 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-08cab540-038f-47eb-9c09-53c740f4cb56 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '797ea6f2ee7b41098addbabc4be4e487'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:07.509754 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-08cab540-038f-47eb-9c09-53c740f4cb56 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '797ea6f2ee7b41098addbabc4be4e487'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:07.519432 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-08cab540-038f-47eb-9c09-53c740f4cb56 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '797ea6f2ee7b41098addbabc4be4e487'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:07.520312 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-08cab540-038f-47eb-9c09-53c740f4cb56 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '797ea6f2ee7b41098addbabc4be4e487'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:07.529126 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-08cab540-038f-47eb-9c09-53c740f4cb56 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '797ea6f2ee7b41098addbabc4be4e487'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:07.548824 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1053/2100] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:57:07 2022] DELETE /identity/v3/users/797ea6f2ee7b41098addbabc4be4e487 => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 00:57:07.772344 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22aa4257-5b75-47c1-86a2-621e9d013998 service neutron] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:07.773973 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-22aa4257-5b75-47c1-86a2-621e9d013998 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:07.786415 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22aa4257-5b75-47c1-86a2-621e9d013998 service neutron] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:07.848807 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a361a9-ac19-4241-a738-a7e38f1854ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:07.849779 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a361a9-ac19-4241-a738-a7e38f1854ad admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:07.850053 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a361a9-ac19-4241-a738-a7e38f1854ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:07.850324 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a361a9-ac19-4241-a738-a7e38f1854ad admin admin] PATH_INFO: `/v3/projects/ba896797e1ca468dbfaeca0b414589c8` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:07.851477 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a361a9-ac19-4241-a738-a7e38f1854ad admin admin] RBAC: Authorizing `identity:delete_project(project_id=ba896797e1ca468dbfaeca0b414589c8)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:07.853430 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a361a9-ac19-4241-a738-a7e38f1854ad admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:07.866917 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:57:07.874753 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08cab540-038f-47eb-9c09-53c740f4cb56 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:07.876164 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-08cab540-038f-47eb-9c09-53c740f4cb56 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:07.891858 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08cab540-038f-47eb-9c09-53c740f4cb56 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:07.904326 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-402715bf-6e43-4b89-b9e5-8145e4a28f73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:07.904968 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402715bf-6e43-4b89-b9e5-8145e4a28f73 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:07.905251 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402715bf-6e43-4b89-b9e5-8145e4a28f73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:07.905547 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402715bf-6e43-4b89-b9e5-8145e4a28f73 admin admin] PATH_INFO: `/v3/projects/d410822d4a0e49029e0d62cab39bbd45` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:07.906678 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-402715bf-6e43-4b89-b9e5-8145e4a28f73 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d410822d4a0e49029e0d62cab39bbd45)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:07.908588 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-402715bf-6e43-4b89-b9e5-8145e4a28f73 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:07.913042 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-a9a361a9-ac19-4241-a738-a7e38f1854ad admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ba896797e1ca468dbfaeca0b414589c8', 'request_id': 'req-a9a361a9-ac19-4241-a738-a7e38f1854ad'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:07.921643 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-a9a361a9-ac19-4241-a738-a7e38f1854ad admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ba896797e1ca468dbfaeca0b414589c8', 'request_id': 'req-a9a361a9-ac19-4241-a738-a7e38f1854ad'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:07.927434 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-a9a361a9-ac19-4241-a738-a7e38f1854ad admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ba896797e1ca468dbfaeca0b414589c8', 'request_id': 'req-a9a361a9-ac19-4241-a738-a7e38f1854ad'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:07.939932 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-a9a361a9-ac19-4241-a738-a7e38f1854ad admin admin] The token cache is being invalidate because project ba896797e1ca468dbfaeca0b414589c8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81386) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:57:07.940217 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-a9a361a9-ac19-4241-a738-a7e38f1854ad admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:07.942445 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-a9a361a9-ac19-4241-a738-a7e38f1854ad admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:07.944286 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1048/2101] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:57:07 2022] DELETE /identity/v3/projects/ba896797e1ca468dbfaeca0b414589c8 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 00:57:07.948462 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a361a9-ac19-4241-a738-a7e38f1854ad admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:07.950262 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-a9a361a9-ac19-4241-a738-a7e38f1854ad admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:07.971573 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-402715bf-6e43-4b89-b9e5-8145e4a28f73 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd410822d4a0e49029e0d62cab39bbd45', 'request_id': 'req-402715bf-6e43-4b89-b9e5-8145e4a28f73'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:07.981041 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-402715bf-6e43-4b89-b9e5-8145e4a28f73 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd410822d4a0e49029e0d62cab39bbd45', 'request_id': 'req-402715bf-6e43-4b89-b9e5-8145e4a28f73'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:07.988920 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-402715bf-6e43-4b89-b9e5-8145e4a28f73 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd410822d4a0e49029e0d62cab39bbd45', 'request_id': 'req-402715bf-6e43-4b89-b9e5-8145e4a28f73'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:08.013240 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-402715bf-6e43-4b89-b9e5-8145e4a28f73 admin admin] The token cache is being invalidate because project d410822d4a0e49029e0d62cab39bbd45 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:57:08.013552 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-402715bf-6e43-4b89-b9e5-8145e4a28f73 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:08.014207 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-402715bf-6e43-4b89-b9e5-8145e4a28f73 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:08.016019 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1054/2102] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:57:07 2022] DELETE /identity/v3/projects/d410822d4a0e49029e0d62cab39bbd45 => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:57:08.020895 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36182fc9-eda1-4287-9642-83f0ccd9a3cd None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:08.021206 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36182fc9-eda1-4287-9642-83f0ccd9a3cd None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:08.022131 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36182fc9-eda1-4287-9642-83f0ccd9a3cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:08.034594 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a361a9-ac19-4241-a738-a7e38f1854ad admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:08.055000 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-36182fc9-eda1-4287-9642-83f0ccd9a3cd None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:57:08.055000 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-36182fc9-eda1-4287-9642-83f0ccd9a3cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:08.096108 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:57:08.166314 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-929fe488-c0c8-45d7-a2a7-4d29c695d8b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:08.166505 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1055/2103] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:57:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:57:08.167377 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929fe488-c0c8-45d7-a2a7-4d29c695d8b0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:08.169160 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929fe488-c0c8-45d7-a2a7-4d29c695d8b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:08.169720 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929fe488-c0c8-45d7-a2a7-4d29c695d8b0 admin admin] PATH_INFO: `/v3/users/89d579de163747f2a56d0353718ecec3` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:08.174051 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-929fe488-c0c8-45d7-a2a7-4d29c695d8b0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=89d579de163747f2a56d0353718ecec3)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:08.176004 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-929fe488-c0c8-45d7-a2a7-4d29c695d8b0 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:08.182543 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f45e79e7-4412-49e7-ad17-28043b8509ec None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:08.182829 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f45e79e7-4412-49e7-ad17-28043b8509ec None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:08.183099 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f45e79e7-4412-49e7-ad17-28043b8509ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:08.205504 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-f45e79e7-4412-49e7-ad17-28043b8509ec None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:57:08.206711 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-f45e79e7-4412-49e7-ad17-28043b8509ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:08.230442 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1056/2104] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:57:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:57:08.235613 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f45e79e7-4412-49e7-ad17-28043b8509ec None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:08.245956 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f45e79e7-4412-49e7-ad17-28043b8509ec None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:08.250494 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-929fe488-c0c8-45d7-a2a7-4d29c695d8b0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '89d579de163747f2a56d0353718ecec3'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:08.251060 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c10494f9-2341-4840-b017-b03e81feeeaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:08.251657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10494f9-2341-4840-b017-b03e81feeeaa admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:08.251930 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10494f9-2341-4840-b017-b03e81feeeaa admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:08.252197 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10494f9-2341-4840-b017-b03e81feeeaa admin admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:08.252863 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c10494f9-2341-4840-b017-b03e81feeeaa admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:57:08.253159 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c10494f9-2341-4840-b017-b03e81feeeaa admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:08.254239 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:57:08.254906 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c10494f9-2341-4840-b017-b03e81feeeaa admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:08.260275 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-929fe488-c0c8-45d7-a2a7-4d29c695d8b0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '89d579de163747f2a56d0353718ecec3'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:08.261415 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-929fe488-c0c8-45d7-a2a7-4d29c695d8b0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '89d579de163747f2a56d0353718ecec3'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:08.278437 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1057/2105] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 00:57:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:57:08.280564 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c10494f9-2341-4840-b017-b03e81feeeaa admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:08.289882 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-929fe488-c0c8-45d7-a2a7-4d29c695d8b0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '89d579de163747f2a56d0353718ecec3'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:08.290647 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-929fe488-c0c8-45d7-a2a7-4d29c695d8b0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '89d579de163747f2a56d0353718ecec3'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:08.294200 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c10494f9-2341-4840-b017-b03e81feeeaa admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:08.300384 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-502166d8-5072-45f6-a020-daf548cf1c63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:08.301170 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-502166d8-5072-45f6-a020-daf548cf1c63 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:08.301490 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-502166d8-5072-45f6-a020-daf548cf1c63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:08.301824 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-502166d8-5072-45f6-a020-daf548cf1c63 admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:08.303545 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-502166d8-5072-45f6-a020-daf548cf1c63 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:08.305523 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-502166d8-5072-45f6-a020-daf548cf1c63 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:08.314579 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1049/2106] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:57:07 2022] DELETE /identity/v3/users/89d579de163747f2a56d0353718ecec3 => generated 0 bytes in 366 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:57:08.339532 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1058/2107] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:57:08 2022] POST /identity/v3/projects => generated 387 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 00:57:08.343635 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-929fe488-c0c8-45d7-a2a7-4d29c695d8b0 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:08.345049 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-929fe488-c0c8-45d7-a2a7-4d29c695d8b0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:08.428107 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-929fe488-c0c8-45d7-a2a7-4d29c695d8b0 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:08.494621 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d349e03e-3f4d-4193-9548-e117f974b9fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:08.495234 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d349e03e-3f4d-4193-9548-e117f974b9fa admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:08.495510 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d349e03e-3f4d-4193-9548-e117f974b9fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:08.495799 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d349e03e-3f4d-4193-9548-e117f974b9fa admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:08.496413 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d349e03e-3f4d-4193-9548-e117f974b9fa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:08.498419 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d349e03e-3f4d-4193-9548-e117f974b9fa admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:08.524693 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1050/2108] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:57:08 2022] POST /identity/v3/users => generated 294 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:57:08.525995 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-502166d8-5072-45f6-a020-daf548cf1c63 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:08.536145 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-502166d8-5072-45f6-a020-daf548cf1c63 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:08.540895 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b65bff-589d-456e-ad54-1c496959ee8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:08.541783 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b65bff-589d-456e-ad54-1c496959ee8f admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:08.542060 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b65bff-589d-456e-ad54-1c496959ee8f admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:08.542326 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b65bff-589d-456e-ad54-1c496959ee8f admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:08.543039 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b65bff-589d-456e-ad54-1c496959ee8f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:08.544572 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b65bff-589d-456e-ad54-1c496959ee8f admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:08.556297 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1059/2109] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:57:08 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:57:08.560453 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d349e03e-3f4d-4193-9548-e117f974b9fa admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:08.573303 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d349e03e-3f4d-4193-9548-e117f974b9fa admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:08.578176 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c0ff669-823f-4776-a514-919e07ae537f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:08.579297 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0ff669-823f-4776-a514-919e07ae537f admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:08.579548 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0ff669-823f-4776-a514-919e07ae537f admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:08.579823 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0ff669-823f-4776-a514-919e07ae537f admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:08.580471 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c0ff669-823f-4776-a514-919e07ae537f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:08.582097 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c0ff669-823f-4776-a514-919e07ae537f admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:08.594787 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1051/2110] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:57:08 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 00:57:08.598946 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b65bff-589d-456e-ad54-1c496959ee8f admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:08.610145 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b65bff-589d-456e-ad54-1c496959ee8f admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:08.618599 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c74dd9c-0539-4fe2-8f62-c025499bda58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:08.619278 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c74dd9c-0539-4fe2-8f62-c025499bda58 admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:08.619578 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c74dd9c-0539-4fe2-8f62-c025499bda58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:08.619850 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c74dd9c-0539-4fe2-8f62-c025499bda58 admin admin] PATH_INFO: `/v3/projects/f5780a6d6dde45ae80cd574620660f24/users/89d411e9a4ed47b3ba00b8d7ced57155/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:08.638655 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c74dd9c-0539-4fe2-8f62-c025499bda58 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f5780a6d6dde45ae80cd574620660f24, user_id=89d411e9a4ed47b3ba00b8d7ced57155, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:08.641788 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c74dd9c-0539-4fe2-8f62-c025499bda58 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:08.643122 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c0ff669-823f-4776-a514-919e07ae537f admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:08.644958 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-1c0ff669-823f-4776-a514-919e07ae537f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:08.659095 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c0ff669-823f-4776-a514-919e07ae537f admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:08.663765 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1060/2111] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:57:08 2022] PUT /identity/v3/projects/f5780a6d6dde45ae80cd574620660f24/users/89d411e9a4ed47b3ba00b8d7ced57155/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:57:08.676288 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-579a5416-c2de-485c-abe4-23b5601567e2 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:08.676435 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-579a5416-c2de-485c-abe4-23b5601567e2 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:08.676729 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-579a5416-c2de-485c-abe4-23b5601567e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:08.707301 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-579a5416-c2de-485c-abe4-23b5601567e2 None None] MFA Rules not processed for user `89d411e9a4ed47b3ba00b8d7ced57155`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:57:08.708669 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-579a5416-c2de-485c-abe4-23b5601567e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:08.714320 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:DELETE=3,keystone:SELECT=94,keystone:INSERT=3 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:57:08.726498 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf438f9-4643-4a4d-b992-1471dc0e4466 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:08.727258 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf438f9-4643-4a4d-b992-1471dc0e4466 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:08.727528 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf438f9-4643-4a4d-b992-1471dc0e4466 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:08.727795 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf438f9-4643-4a4d-b992-1471dc0e4466 admin admin] PATH_INFO: `/v3/projects/1fba20633c144fe0a8572d2a2343f8c9` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:08.728981 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cf438f9-4643-4a4d-b992-1471dc0e4466 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1fba20633c144fe0a8572d2a2343f8c9)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:08.731221 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cf438f9-4643-4a4d-b992-1471dc0e4466 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:08.787686 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1061/2112] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:57:08 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:57:08.792317 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-579a5416-c2de-485c-abe4-23b5601567e2 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:08.792473 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-7cf438f9-4643-4a4d-b992-1471dc0e4466 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1fba20633c144fe0a8572d2a2343f8c9', 'request_id': 'req-7cf438f9-4643-4a4d-b992-1471dc0e4466'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:08.800352 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-7cf438f9-4643-4a4d-b992-1471dc0e4466 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1fba20633c144fe0a8572d2a2343f8c9', 'request_id': 'req-7cf438f9-4643-4a4d-b992-1471dc0e4466'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:08.803292 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-579a5416-c2de-485c-abe4-23b5601567e2 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:08.808503 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-7cf438f9-4643-4a4d-b992-1471dc0e4466 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1fba20633c144fe0a8572d2a2343f8c9', 'request_id': 'req-7cf438f9-4643-4a4d-b992-1471dc0e4466'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:08.823585 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-7cf438f9-4643-4a4d-b992-1471dc0e4466 admin admin] The token cache is being invalidate because project 1fba20633c144fe0a8572d2a2343f8c9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81386) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:57:08.823992 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-7cf438f9-4643-4a4d-b992-1471dc0e4466 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:08.825038 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-7cf438f9-4643-4a4d-b992-1471dc0e4466 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:08.826806 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1052/2113] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:57:08 2022] DELETE /identity/v3/projects/1fba20633c144fe0a8572d2a2343f8c9 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:57:08.831270 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf438f9-4643-4a4d-b992-1471dc0e4466 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:08.832710 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-7cf438f9-4643-4a4d-b992-1471dc0e4466 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:08.851694 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:57:08.866664 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f2cee2b-b636-48bb-9cfe-7fd8710c03b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:08.867275 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2cee2b-b636-48bb-9cfe-7fd8710c03b6 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:08.867675 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2cee2b-b636-48bb-9cfe-7fd8710c03b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:08.867947 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2cee2b-b636-48bb-9cfe-7fd8710c03b6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:08.868577 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f2cee2b-b636-48bb-9cfe-7fd8710c03b6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:08.870499 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f2cee2b-b636-48bb-9cfe-7fd8710c03b6 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:08.900244 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1062/2114] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:57:08 2022] POST /identity/v3/projects => generated 387 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 00:57:08.903729 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf438f9-4643-4a4d-b992-1471dc0e4466 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:08.904301 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f2cee2b-b636-48bb-9cfe-7fd8710c03b6 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:08.906388 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-0f2cee2b-b636-48bb-9cfe-7fd8710c03b6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:08.966666 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c99ac594-1a68-4083-82f0-cd997b7fbb17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:08.967210 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99ac594-1a68-4083-82f0-cd997b7fbb17 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:08.967489 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99ac594-1a68-4083-82f0-cd997b7fbb17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:08.967758 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99ac594-1a68-4083-82f0-cd997b7fbb17 admin admin] PATH_INFO: `/v3/users/f2b6bf09fc3846e6ad42b0e406c45e7f` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:08.968922 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c99ac594-1a68-4083-82f0-cd997b7fbb17 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f2b6bf09fc3846e6ad42b0e406c45e7f)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:08.970979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c99ac594-1a68-4083-82f0-cd997b7fbb17 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:08.979749 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f2cee2b-b636-48bb-9cfe-7fd8710c03b6 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:08.985351 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1f100c-5f95-48b8-81cc-9b049c567b4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:08.985968 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1f100c-5f95-48b8-81cc-9b049c567b4c admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:08.986247 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1f100c-5f95-48b8-81cc-9b049c567b4c admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:08.986764 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1f100c-5f95-48b8-81cc-9b049c567b4c admin admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:08.987284 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f1f100c-5f95-48b8-81cc-9b049c567b4c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:08.990170 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f1f100c-5f95-48b8-81cc-9b049c567b4c admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:09.010586 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:57:09.015038 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1063/2115] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:57:08 2022] POST /identity/v3/users => generated 300 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 00:57:09.018325 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1f100c-5f95-48b8-81cc-9b049c567b4c admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:09.034690 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1f100c-5f95-48b8-81cc-9b049c567b4c admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:09.037957 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-c99ac594-1a68-4083-82f0-cd997b7fbb17 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f2b6bf09fc3846e6ad42b0e406c45e7f'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:09.040066 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1077ccba-d06d-45c0-b5ca-57c262fdabbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:09.040664 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1077ccba-d06d-45c0-b5ca-57c262fdabbf admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:09.040940 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1077ccba-d06d-45c0-b5ca-57c262fdabbf admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:09.041208 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1077ccba-d06d-45c0-b5ca-57c262fdabbf admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:09.041900 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1077ccba-d06d-45c0-b5ca-57c262fdabbf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:09.043759 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1077ccba-d06d-45c0-b5ca-57c262fdabbf admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:09.046125 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-c99ac594-1a68-4083-82f0-cd997b7fbb17 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f2b6bf09fc3846e6ad42b0e406c45e7f'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:09.046929 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-c99ac594-1a68-4083-82f0-cd997b7fbb17 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f2b6bf09fc3846e6ad42b0e406c45e7f'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:09.055442 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-c99ac594-1a68-4083-82f0-cd997b7fbb17 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f2b6bf09fc3846e6ad42b0e406c45e7f'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:09.055939 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1064/2116] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:57:09 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 00:57:09.056223 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-c99ac594-1a68-4083-82f0-cd997b7fbb17 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f2b6bf09fc3846e6ad42b0e406c45e7f'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:09.060141 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1077ccba-d06d-45c0-b5ca-57c262fdabbf admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:09.061987 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-1077ccba-d06d-45c0-b5ca-57c262fdabbf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:09.074733 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1053/2117] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:57:08 2022] DELETE /identity/v3/users/f2b6bf09fc3846e6ad42b0e406c45e7f => generated 0 bytes in 244 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:57:09.075037 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1077ccba-d06d-45c0-b5ca-57c262fdabbf admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:09.135120 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-737b70c4-03f1-469a-8712-2fbd7d13e9d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:09.135814 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737b70c4-03f1-469a-8712-2fbd7d13e9d0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:09.136089 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737b70c4-03f1-469a-8712-2fbd7d13e9d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:09.136358 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737b70c4-03f1-469a-8712-2fbd7d13e9d0 admin admin] PATH_INFO: `/v3/projects/9802a2faa6b749f9b4850348c95f8ea3/users/f99303f3d1b646398bdf8e7a4bd1cfc9/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:09.152433 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-737b70c4-03f1-469a-8712-2fbd7d13e9d0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9802a2faa6b749f9b4850348c95f8ea3, user_id=f99303f3d1b646398bdf8e7a4bd1cfc9, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:09.155109 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-737b70c4-03f1-469a-8712-2fbd7d13e9d0 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:09.168951 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1065/2118] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:57:09 2022] PUT /identity/v3/projects/9802a2faa6b749f9b4850348c95f8ea3/users/f99303f3d1b646398bdf8e7a4bd1cfc9/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:57:09.172916 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c99ac594-1a68-4083-82f0-cd997b7fbb17 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:09.186202 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c99ac594-1a68-4083-82f0-cd997b7fbb17 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:09.278062 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-225e28d3-bb74-4923-b7e9-51e76e2d4432 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:09.278832 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225e28d3-bb74-4923-b7e9-51e76e2d4432 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:09.279114 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225e28d3-bb74-4923-b7e9-51e76e2d4432 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:09.279535 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225e28d3-bb74-4923-b7e9-51e76e2d4432 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:09.280269 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-225e28d3-bb74-4923-b7e9-51e76e2d4432 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:09.281915 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-225e28d3-bb74-4923-b7e9-51e76e2d4432 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:09.296982 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1054/2119] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:57:09 2022] GET /identity/v3/roles => generated 1420 bytes in 122 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 00:57:09.298159 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-737b70c4-03f1-469a-8712-2fbd7d13e9d0 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:09.313489 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-737b70c4-03f1-469a-8712-2fbd7d13e9d0 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:09.318919 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4f3cae-0b62-4fd1-b820-d0fda0ed84d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:09.319574 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4f3cae-0b62-4fd1-b820-d0fda0ed84d9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:09.319849 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4f3cae-0b62-4fd1-b820-d0fda0ed84d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:09.320115 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4f3cae-0b62-4fd1-b820-d0fda0ed84d9 admin admin] PATH_INFO: `/v3/projects/9802a2faa6b749f9b4850348c95f8ea3/users/f99303f3d1b646398bdf8e7a4bd1cfc9/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:09.322059 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a4f3cae-0b62-4fd1-b820-d0fda0ed84d9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9802a2faa6b749f9b4850348c95f8ea3, user_id=f99303f3d1b646398bdf8e7a4bd1cfc9, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:09.324688 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a4f3cae-0b62-4fd1-b820-d0fda0ed84d9 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:09.336630 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1066/2120] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:57:09 2022] PUT /identity/v3/projects/9802a2faa6b749f9b4850348c95f8ea3/users/f99303f3d1b646398bdf8e7a4bd1cfc9/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:57:09.347449 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f136f356-ce60-4e46-ab8d-1f5bba2c1f49 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:09.347730 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f136f356-ce60-4e46-ab8d-1f5bba2c1f49 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:09.347995 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f136f356-ce60-4e46-ab8d-1f5bba2c1f49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:09.378741 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-f136f356-ce60-4e46-ab8d-1f5bba2c1f49 None None] MFA Rules not processed for user `f99303f3d1b646398bdf8e7a4bd1cfc9`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:57:09.380087 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-f136f356-ce60-4e46-ab8d-1f5bba2c1f49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:09.497701 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1055/2121] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:57:09 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:57:09.517685 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4f3cae-0b62-4fd1-b820-d0fda0ed84d9 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:09.519425 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-7a4f3cae-0b62-4fd1-b820-d0fda0ed84d9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:09.545325 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:57:09.610089 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4f3cae-0b62-4fd1-b820-d0fda0ed84d9 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:09.615922 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f136f356-ce60-4e46-ab8d-1f5bba2c1f49 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:09.617406 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-f136f356-ce60-4e46-ab8d-1f5bba2c1f49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:09.619111 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82553ceb-814c-40e9-afcf-b2f00e15df49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:09.621380 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82553ceb-814c-40e9-afcf-b2f00e15df49 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:09.621895 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82553ceb-814c-40e9-afcf-b2f00e15df49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:09.622530 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82553ceb-814c-40e9-afcf-b2f00e15df49 admin admin] PATH_INFO: `/v3/projects/bec11bdddb3e4993bbbdf5042faa8a60` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:09.624273 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82553ceb-814c-40e9-afcf-b2f00e15df49 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bec11bdddb3e4993bbbdf5042faa8a60)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:09.626453 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82553ceb-814c-40e9-afcf-b2f00e15df49 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:09.662814 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:57:09.686963 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-82553ceb-814c-40e9-afcf-b2f00e15df49 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bec11bdddb3e4993bbbdf5042faa8a60', 'request_id': 'req-82553ceb-814c-40e9-afcf-b2f00e15df49'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:09.700588 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-82553ceb-814c-40e9-afcf-b2f00e15df49 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bec11bdddb3e4993bbbdf5042faa8a60', 'request_id': 'req-82553ceb-814c-40e9-afcf-b2f00e15df49'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:09.709145 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-82553ceb-814c-40e9-afcf-b2f00e15df49 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bec11bdddb3e4993bbbdf5042faa8a60', 'request_id': 'req-82553ceb-814c-40e9-afcf-b2f00e15df49'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:09.738129 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-82553ceb-814c-40e9-afcf-b2f00e15df49 admin admin] The token cache is being invalidate because project bec11bdddb3e4993bbbdf5042faa8a60 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:57:09.738748 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-82553ceb-814c-40e9-afcf-b2f00e15df49 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:09.739783 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-82553ceb-814c-40e9-afcf-b2f00e15df49 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:09.748220 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f136f356-ce60-4e46-ab8d-1f5bba2c1f49 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:09.749062 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1067/2122] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:57:09 2022] DELETE /identity/v3/projects/bec11bdddb3e4993bbbdf5042faa8a60 => generated 0 bytes in 225 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:57:09.752182 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ee14b5f-055a-4981-8229-476a613c767a None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:09.752759 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ee14b5f-055a-4981-8229-476a613c767a None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:09.753424 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ee14b5f-055a-4981-8229-476a613c767a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:09.791373 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-3ee14b5f-055a-4981-8229-476a613c767a None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:57:09.792624 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-3ee14b5f-055a-4981-8229-476a613c767a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:09.830929 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ce80f7-fd16-4630-a3df-a9530626bd80 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:09.831558 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ce80f7-fd16-4630-a3df-a9530626bd80 service cinder] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:09.832431 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ce80f7-fd16-4630-a3df-a9530626bd80 service cinder] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:09.832598 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ce80f7-fd16-4630-a3df-a9530626bd80 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:09.835072 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-72ce80f7-fd16-4630-a3df-a9530626bd80 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:09.901804 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1068/2123] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:57:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:57:09.913372 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0dd5fa5-2dc3-46d3-9705-43e655ead488 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:09.913982 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0dd5fa5-2dc3-46d3-9705-43e655ead488 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:09.915192 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0dd5fa5-2dc3-46d3-9705-43e655ead488 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:09.935635 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ce80f7-fd16-4630-a3df-a9530626bd80 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:09.937477 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ce80f7-fd16-4630-a3df-a9530626bd80 service cinder] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:09.940427 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-b0dd5fa5-2dc3-46d3-9705-43e655ead488 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:57:09.941685 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-b0dd5fa5-2dc3-46d3-9705-43e655ead488 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:09.948902 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1056/2124] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:57:09 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 334 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 00:57:09.953294 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1069/2125] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:57:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:57:09.958499 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ce80f7-fd16-4630-a3df-a9530626bd80 service cinder] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:09.971219 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ce80f7-fd16-4630-a3df-a9530626bd80 service cinder] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:09.981347 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24eec8fd-3f6a-412b-8800-fb6b91774646 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:09.981887 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24eec8fd-3f6a-412b-8800-fb6b91774646 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:09.982159 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24eec8fd-3f6a-412b-8800-fb6b91774646 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:09.982555 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24eec8fd-3f6a-412b-8800-fb6b91774646 admin admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:09.983334 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24eec8fd-3f6a-412b-8800-fb6b91774646 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:57:09.983741 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24eec8fd-3f6a-412b-8800-fb6b91774646 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:09.985987 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24eec8fd-3f6a-412b-8800-fb6b91774646 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:10.015118 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1057/2126] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 00:57:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:57:10.017272 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0dd5fa5-2dc3-46d3-9705-43e655ead488 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:10.033434 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0dd5fa5-2dc3-46d3-9705-43e655ead488 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:10.044221 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c5d5a1-8449-4de9-b05f-02fd72e4d03e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:10.045094 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c5d5a1-8449-4de9-b05f-02fd72e4d03e admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:10.045676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c5d5a1-8449-4de9-b05f-02fd72e4d03e admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:10.046336 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c5d5a1-8449-4de9-b05f-02fd72e4d03e admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:10.047640 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c5d5a1-8449-4de9-b05f-02fd72e4d03e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:10.050292 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c5d5a1-8449-4de9-b05f-02fd72e4d03e admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:10.088579 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1070/2127] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:57:10 2022] POST /identity/v3/projects => generated 369 bytes in 72 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:57:10.092584 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24eec8fd-3f6a-412b-8800-fb6b91774646 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:10.109417 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24eec8fd-3f6a-412b-8800-fb6b91774646 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:10.140288 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c5d5a1-8449-4de9-b05f-02fd72e4d03e admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:10.142718 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-d4c5d5a1-8449-4de9-b05f-02fd72e4d03e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:10.174625 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:57:10.186509 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a7ff03-280b-4c68-9b5c-aa262f3db54d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:10.187400 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a7ff03-280b-4c68-9b5c-aa262f3db54d admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:10.187790 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a7ff03-280b-4c68-9b5c-aa262f3db54d admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:10.188076 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a7ff03-280b-4c68-9b5c-aa262f3db54d admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:10.188711 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01a7ff03-280b-4c68-9b5c-aa262f3db54d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:10.190762 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01a7ff03-280b-4c68-9b5c-aa262f3db54d admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:10.217497 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1058/2128] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:57:10 2022] POST /identity/v3/users => generated 285 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 00:57:10.218561 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a7ff03-280b-4c68-9b5c-aa262f3db54d admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:10.226694 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c5d5a1-8449-4de9-b05f-02fd72e4d03e admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:10.231091 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a7ff03-280b-4c68-9b5c-aa262f3db54d admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:10.231553 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6854da9b-6ad8-4921-871a-c7a53a600623 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:10.232184 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6854da9b-6ad8-4921-871a-c7a53a600623 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:10.232461 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6854da9b-6ad8-4921-871a-c7a53a600623 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:10.232727 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6854da9b-6ad8-4921-871a-c7a53a600623 admin admin] PATH_INFO: `/v3/users/89d411e9a4ed47b3ba00b8d7ced57155` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:10.233884 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6854da9b-6ad8-4921-871a-c7a53a600623 admin admin] RBAC: Authorizing `identity:delete_user(user_id=89d411e9a4ed47b3ba00b8d7ced57155)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:10.235786 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6854da9b-6ad8-4921-871a-c7a53a600623 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:10.236826 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a4dde4-0f5c-47dd-83a3-9db6647fe830 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:10.237918 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a4dde4-0f5c-47dd-83a3-9db6647fe830 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:10.238004 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a4dde4-0f5c-47dd-83a3-9db6647fe830 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:10.238319 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a4dde4-0f5c-47dd-83a3-9db6647fe830 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:10.239249 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3a4dde4-0f5c-47dd-83a3-9db6647fe830 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:10.246917 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3a4dde4-0f5c-47dd-83a3-9db6647fe830 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:10.272218 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1059/2129] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:57:10 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:57:10.273549 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a4dde4-0f5c-47dd-83a3-9db6647fe830 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:10.281323 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:57:10.287669 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a4dde4-0f5c-47dd-83a3-9db6647fe830 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:10.294423 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c67e234-21a8-45f5-bf34-063e420b64e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:10.295339 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c67e234-21a8-45f5-bf34-063e420b64e4 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:10.295767 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c67e234-21a8-45f5-bf34-063e420b64e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:10.296042 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c67e234-21a8-45f5-bf34-063e420b64e4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:10.296656 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-6854da9b-6ad8-4921-871a-c7a53a600623 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '89d411e9a4ed47b3ba00b8d7ced57155'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:10.296789 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c67e234-21a8-45f5-bf34-063e420b64e4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:10.297349 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-6854da9b-6ad8-4921-871a-c7a53a600623 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '89d411e9a4ed47b3ba00b8d7ced57155'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:10.298521 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c67e234-21a8-45f5-bf34-063e420b64e4 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:10.310628 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1060/2130] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:57:10 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:57:10.314368 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c67e234-21a8-45f5-bf34-063e420b64e4 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:10.321422 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-6854da9b-6ad8-4921-871a-c7a53a600623 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '89d411e9a4ed47b3ba00b8d7ced57155'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:10.322127 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-6854da9b-6ad8-4921-871a-c7a53a600623 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '89d411e9a4ed47b3ba00b8d7ced57155'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:10.326776 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c67e234-21a8-45f5-bf34-063e420b64e4 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:10.331893 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d68f762f-4707-4a4a-9725-fcc66e5dad1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:10.332556 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68f762f-4707-4a4a-9725-fcc66e5dad1c admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:10.332874 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68f762f-4707-4a4a-9725-fcc66e5dad1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:10.333126 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68f762f-4707-4a4a-9725-fcc66e5dad1c admin admin] PATH_INFO: `/v3/projects/4e348390b8774830a7542bb983bd5672/users/1dd2549f4f29404f92abbdc086d48ac7/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:10.333231 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-6854da9b-6ad8-4921-871a-c7a53a600623 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '89d411e9a4ed47b3ba00b8d7ced57155'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:10.353550 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1071/2131] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:57:10 2022] DELETE /identity/v3/users/89d411e9a4ed47b3ba00b8d7ced57155 => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 00:57:10.357889 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa06e852-ddef-4d95-8c2b-5a89b5b94a18 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:10.358175 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa06e852-ddef-4d95-8c2b-5a89b5b94a18 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:10.358453 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa06e852-ddef-4d95-8c2b-5a89b5b94a18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:10.362595 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d68f762f-4707-4a4a-9725-fcc66e5dad1c admin admin] RBAC: Authorizing `identity:create_grant(project_id=4e348390b8774830a7542bb983bd5672, user_id=1dd2549f4f29404f92abbdc086d48ac7, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:10.366890 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d68f762f-4707-4a4a-9725-fcc66e5dad1c admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:10.381617 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-fa06e852-ddef-4d95-8c2b-5a89b5b94a18 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:57:10.382882 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-fa06e852-ddef-4d95-8c2b-5a89b5b94a18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:10.391251 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1061/2132] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:57:10 2022] PUT /identity/v3/projects/4e348390b8774830a7542bb983bd5672/users/1dd2549f4f29404f92abbdc086d48ac7/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 77 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:57:10.403265 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f809fa44-9438-48f0-a092-3035a91f7b57 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:10.403544 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f809fa44-9438-48f0-a092-3035a91f7b57 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:10.403807 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f809fa44-9438-48f0-a092-3035a91f7b57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:10.436114 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-f809fa44-9438-48f0-a092-3035a91f7b57 None None] MFA Rules not processed for user `1dd2549f4f29404f92abbdc086d48ac7`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:57:10.437431 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-f809fa44-9438-48f0-a092-3035a91f7b57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:10.445548 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1072/2133] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:57:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 00:57:10.455486 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa06e852-ddef-4d95-8c2b-5a89b5b94a18 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:10.456928 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-fa06e852-ddef-4d95-8c2b-5a89b5b94a18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:10.506864 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:57:10.519805 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa06e852-ddef-4d95-8c2b-5a89b5b94a18 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:10.520311 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1062/2134] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:57:10 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 00:57:10.526788 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9ed4bc-bf8a-45ad-b151-1940348b4de4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:10.527390 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9ed4bc-bf8a-45ad-b151-1940348b4de4 service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:10.527670 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9ed4bc-bf8a-45ad-b151-1940348b4de4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:10.527941 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9ed4bc-bf8a-45ad-b151-1940348b4de4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:10.534468 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f809fa44-9438-48f0-a092-3035a91f7b57 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:10.536075 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-f809fa44-9438-48f0-a092-3035a91f7b57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:10.541987 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f9ed4bc-bf8a-45ad-b151-1940348b4de4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:10.543669 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f9ed4bc-bf8a-45ad-b151-1940348b4de4 service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:10.555831 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1073/2135] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:57:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 00:57:10.601334 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f809fa44-9438-48f0-a092-3035a91f7b57 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:10.605529 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79cf46e-a83e-4339-b1a0-3cfe625e14da service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4ebfc0abf1b842459a44ca9324252688', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:10.606144 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79cf46e-a83e-4339-b1a0-3cfe625e14da service glance] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:10.606421 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79cf46e-a83e-4339-b1a0-3cfe625e14da service glance] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:10.606686 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79cf46e-a83e-4339-b1a0-3cfe625e14da service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:10.616913 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e79cf46e-a83e-4339-b1a0-3cfe625e14da service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:10.618804 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e79cf46e-a83e-4339-b1a0-3cfe625e14da service glance] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:10.634579 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1063/2136] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:57:10 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:57:10.878742 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9ed4bc-bf8a-45ad-b151-1940348b4de4 service neutron] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:10.881805 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-9f9ed4bc-bf8a-45ad-b151-1940348b4de4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:10.900882 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9ed4bc-bf8a-45ad-b151-1940348b4de4 service neutron] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:10.908294 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6484667c-2dd1-43ca-aaff-f313980cfe20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:10.908960 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6484667c-2dd1-43ca-aaff-f313980cfe20 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:10.909201 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6484667c-2dd1-43ca-aaff-f313980cfe20 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:10.909526 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6484667c-2dd1-43ca-aaff-f313980cfe20 admin admin] PATH_INFO: `/v3/projects/f5780a6d6dde45ae80cd574620660f24` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:10.912480 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6484667c-2dd1-43ca-aaff-f313980cfe20 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f5780a6d6dde45ae80cd574620660f24)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:10.914928 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6484667c-2dd1-43ca-aaff-f313980cfe20 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:10.972188 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-6484667c-2dd1-43ca-aaff-f313980cfe20 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f5780a6d6dde45ae80cd574620660f24', 'request_id': 'req-6484667c-2dd1-43ca-aaff-f313980cfe20'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:10.983584 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-6484667c-2dd1-43ca-aaff-f313980cfe20 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f5780a6d6dde45ae80cd574620660f24', 'request_id': 'req-6484667c-2dd1-43ca-aaff-f313980cfe20'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:10.990459 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-6484667c-2dd1-43ca-aaff-f313980cfe20 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f5780a6d6dde45ae80cd574620660f24', 'request_id': 'req-6484667c-2dd1-43ca-aaff-f313980cfe20'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:11.006166 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-6484667c-2dd1-43ca-aaff-f313980cfe20 admin admin] The token cache is being invalidate because project f5780a6d6dde45ae80cd574620660f24 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:57:11.006458 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-6484667c-2dd1-43ca-aaff-f313980cfe20 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:11.007554 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-6484667c-2dd1-43ca-aaff-f313980cfe20 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:11.009823 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1074/2137] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:57:10 2022] DELETE /identity/v3/projects/f5780a6d6dde45ae80cd574620660f24 => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:57:11.018254 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79cf46e-a83e-4339-b1a0-3cfe625e14da service glance] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:11.030740 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79cf46e-a83e-4339-b1a0-3cfe625e14da service glance] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:11.126918 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86fcd2f6-f2c3-41ef-a7ea-eb09e9b7be32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:11.127574 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86fcd2f6-f2c3-41ef-a7ea-eb09e9b7be32 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:11.127853 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86fcd2f6-f2c3-41ef-a7ea-eb09e9b7be32 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:11.128121 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86fcd2f6-f2c3-41ef-a7ea-eb09e9b7be32 admin admin] PATH_INFO: `/v3/users/f99303f3d1b646398bdf8e7a4bd1cfc9` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:11.129245 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86fcd2f6-f2c3-41ef-a7ea-eb09e9b7be32 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f99303f3d1b646398bdf8e7a4bd1cfc9)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:11.132375 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86fcd2f6-f2c3-41ef-a7ea-eb09e9b7be32 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:11.134145 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:57:11.232023 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-86fcd2f6-f2c3-41ef-a7ea-eb09e9b7be32 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f99303f3d1b646398bdf8e7a4bd1cfc9'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:11.243669 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-86fcd2f6-f2c3-41ef-a7ea-eb09e9b7be32 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f99303f3d1b646398bdf8e7a4bd1cfc9'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:11.244279 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-86fcd2f6-f2c3-41ef-a7ea-eb09e9b7be32 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f99303f3d1b646398bdf8e7a4bd1cfc9'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:11.252117 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-86fcd2f6-f2c3-41ef-a7ea-eb09e9b7be32 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f99303f3d1b646398bdf8e7a4bd1cfc9'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:11.252642 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-86fcd2f6-f2c3-41ef-a7ea-eb09e9b7be32 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f99303f3d1b646398bdf8e7a4bd1cfc9'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:11.272114 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1064/2138] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:57:11 2022] DELETE /identity/v3/users/f99303f3d1b646398bdf8e7a4bd1cfc9 => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:57:11.653499 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6484667c-2dd1-43ca-aaff-f313980cfe20 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:11.654952 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-6484667c-2dd1-43ca-aaff-f313980cfe20 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:11.681732 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6484667c-2dd1-43ca-aaff-f313980cfe20 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:11.773135 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d19b2c-8f56-4a01-8496-5b078db9ce7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:11.773712 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d19b2c-8f56-4a01-8496-5b078db9ce7e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:11.773992 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d19b2c-8f56-4a01-8496-5b078db9ce7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:11.774255 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d19b2c-8f56-4a01-8496-5b078db9ce7e admin admin] PATH_INFO: `/v3/projects/9802a2faa6b749f9b4850348c95f8ea3` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:11.775375 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3d19b2c-8f56-4a01-8496-5b078db9ce7e admin admin] RBAC: Authorizing `identity:delete_project(project_id=9802a2faa6b749f9b4850348c95f8ea3)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:11.777293 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3d19b2c-8f56-4a01-8496-5b078db9ce7e admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:11.827508 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-e3d19b2c-8f56-4a01-8496-5b078db9ce7e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9802a2faa6b749f9b4850348c95f8ea3', 'request_id': 'req-e3d19b2c-8f56-4a01-8496-5b078db9ce7e'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:11.835820 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-e3d19b2c-8f56-4a01-8496-5b078db9ce7e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9802a2faa6b749f9b4850348c95f8ea3', 'request_id': 'req-e3d19b2c-8f56-4a01-8496-5b078db9ce7e'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:11.841982 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-e3d19b2c-8f56-4a01-8496-5b078db9ce7e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9802a2faa6b749f9b4850348c95f8ea3', 'request_id': 'req-e3d19b2c-8f56-4a01-8496-5b078db9ce7e'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:11.849470 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:57:11.854889 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-e3d19b2c-8f56-4a01-8496-5b078db9ce7e admin admin] The token cache is being invalidate because project 9802a2faa6b749f9b4850348c95f8ea3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:57:11.856563 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-e3d19b2c-8f56-4a01-8496-5b078db9ce7e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:11.856563 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-e3d19b2c-8f56-4a01-8496-5b078db9ce7e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:11.862374 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1075/2139] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:57:11 2022] DELETE /identity/v3/projects/9802a2faa6b749f9b4850348c95f8ea3 => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:57:11.864853 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99fb5679-9642-42a8-8c75-8b2975118b5e None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:11.865606 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99fb5679-9642-42a8-8c75-8b2975118b5e None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:11.866001 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99fb5679-9642-42a8-8c75-8b2975118b5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:11.889684 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-99fb5679-9642-42a8-8c75-8b2975118b5e None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:57:11.890946 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-99fb5679-9642-42a8-8c75-8b2975118b5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:11.952725 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1065/2140] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:57:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:57:11.961870 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a057ee3-3331-47f7-ad14-4f81624b2db1 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:11.962380 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a057ee3-3331-47f7-ad14-4f81624b2db1 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:11.962486 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a057ee3-3331-47f7-ad14-4f81624b2db1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:11.983567 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-1a057ee3-3331-47f7-ad14-4f81624b2db1 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:57:11.984761 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-1a057ee3-3331-47f7-ad14-4f81624b2db1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:11.993775 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1076/2141] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:57:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:57:11.998200 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99fb5679-9642-42a8-8c75-8b2975118b5e None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:12.007870 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99fb5679-9642-42a8-8c75-8b2975118b5e None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:12.012855 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf3f6dd2-d17a-4bf5-982b-1723fc2006b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:12.013464 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3f6dd2-d17a-4bf5-982b-1723fc2006b1 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:12.013738 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3f6dd2-d17a-4bf5-982b-1723fc2006b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:12.014006 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3f6dd2-d17a-4bf5-982b-1723fc2006b1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:12.014649 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf3f6dd2-d17a-4bf5-982b-1723fc2006b1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:57:12.015112 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf3f6dd2-d17a-4bf5-982b-1723fc2006b1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:12.016820 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf3f6dd2-d17a-4bf5-982b-1723fc2006b1 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:12.028984 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1066/2142] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 00:57:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:57:12.033381 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a057ee3-3331-47f7-ad14-4f81624b2db1 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:12.043854 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a057ee3-3331-47f7-ad14-4f81624b2db1 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:12.049250 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6c1f119-a1c3-4a1c-97a2-0ea8cc634ac9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:12.049899 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c1f119-a1c3-4a1c-97a2-0ea8cc634ac9 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:12.050177 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c1f119-a1c3-4a1c-97a2-0ea8cc634ac9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:12.050446 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c1f119-a1c3-4a1c-97a2-0ea8cc634ac9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:12.051124 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6c1f119-a1c3-4a1c-97a2-0ea8cc634ac9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:12.053885 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6c1f119-a1c3-4a1c-97a2-0ea8cc634ac9 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:12.084490 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1077/2143] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:57:12 2022] POST /identity/v3/projects => generated 383 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:57:12.088615 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf3f6dd2-d17a-4bf5-982b-1723fc2006b1 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:12.099723 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf3f6dd2-d17a-4bf5-982b-1723fc2006b1 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:12.159663 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5630eae4-9d0a-40b2-a34c-e00a0b3d4479 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:12.160424 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5630eae4-9d0a-40b2-a34c-e00a0b3d4479 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:12.160616 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5630eae4-9d0a-40b2-a34c-e00a0b3d4479 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:12.160997 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5630eae4-9d0a-40b2-a34c-e00a0b3d4479 admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:12.162473 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5630eae4-9d0a-40b2-a34c-e00a0b3d4479 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:12.164795 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5630eae4-9d0a-40b2-a34c-e00a0b3d4479 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:12.176514 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:57:12.191862 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1067/2144] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:57:12 2022] POST /identity/v3/users => generated 292 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 00:57:12.195728 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6c1f119-a1c3-4a1c-97a2-0ea8cc634ac9 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:12.206774 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6c1f119-a1c3-4a1c-97a2-0ea8cc634ac9 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:12.211783 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bd2b5ef-bf52-47d7-a0e9-a9a8d8451f59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:12.212388 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd2b5ef-bf52-47d7-a0e9-a9a8d8451f59 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:12.212668 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd2b5ef-bf52-47d7-a0e9-a9a8d8451f59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:12.212931 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd2b5ef-bf52-47d7-a0e9-a9a8d8451f59 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:12.213649 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bd2b5ef-bf52-47d7-a0e9-a9a8d8451f59 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:12.215442 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bd2b5ef-bf52-47d7-a0e9-a9a8d8451f59 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:12.232332 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1078/2145] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:57:12 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:57:12.233772 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5630eae4-9d0a-40b2-a34c-e00a0b3d4479 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:12.248731 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5630eae4-9d0a-40b2-a34c-e00a0b3d4479 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:12.252897 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a8eddf-3af3-46fd-8162-66773344cd45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:12.253664 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a8eddf-3af3-46fd-8162-66773344cd45 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:12.254030 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a8eddf-3af3-46fd-8162-66773344cd45 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:12.254309 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a8eddf-3af3-46fd-8162-66773344cd45 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:12.255076 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1a8eddf-3af3-46fd-8162-66773344cd45 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:12.256774 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1a8eddf-3af3-46fd-8162-66773344cd45 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:12.267814 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1068/2146] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:57:12 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:57:12.272731 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bd2b5ef-bf52-47d7-a0e9-a9a8d8451f59 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:12.283191 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bd2b5ef-bf52-47d7-a0e9-a9a8d8451f59 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:12.288632 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d7522a-a97f-4b61-9d8b-37d30fdf679e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:12.289306 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d7522a-a97f-4b61-9d8b-37d30fdf679e admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:12.289587 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d7522a-a97f-4b61-9d8b-37d30fdf679e admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:12.289857 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d7522a-a97f-4b61-9d8b-37d30fdf679e admin admin] PATH_INFO: `/v3/projects/7bc12e750566459ebac4a268b257f1c7/users/71f43ed0cee749b9be1df93f216157b8/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:12.305774 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40d7522a-a97f-4b61-9d8b-37d30fdf679e admin admin] RBAC: Authorizing `identity:create_grant(project_id=7bc12e750566459ebac4a268b257f1c7, user_id=71f43ed0cee749b9be1df93f216157b8, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:12.308420 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40d7522a-a97f-4b61-9d8b-37d30fdf679e admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:12.321793 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1079/2147] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:57:12 2022] PUT /identity/v3/projects/7bc12e750566459ebac4a268b257f1c7/users/71f43ed0cee749b9be1df93f216157b8/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 00:57:12.332002 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97dcbbe8-4a55-40d5-b740-b22dda65144f None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:12.332283 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97dcbbe8-4a55-40d5-b740-b22dda65144f None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:12.332554 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97dcbbe8-4a55-40d5-b740-b22dda65144f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:12.361647 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-97dcbbe8-4a55-40d5-b740-b22dda65144f None None] MFA Rules not processed for user `71f43ed0cee749b9be1df93f216157b8`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:57:12.363189 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-97dcbbe8-4a55-40d5-b740-b22dda65144f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:12.434221 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1069/2148] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:57:12 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:57:12.445368 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d7522a-a97f-4b61-9d8b-37d30fdf679e admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:12.447154 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-40d7522a-a97f-4b61-9d8b-37d30fdf679e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:12.513856 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d7522a-a97f-4b61-9d8b-37d30fdf679e admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:12.517510 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2276acd1-312e-4cd6-976a-f867627a5ed3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:12.518113 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2276acd1-312e-4cd6-976a-f867627a5ed3 service cinder] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:12.518383 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2276acd1-312e-4cd6-976a-f867627a5ed3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:12.518646 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2276acd1-312e-4cd6-976a-f867627a5ed3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:12.528099 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2276acd1-312e-4cd6-976a-f867627a5ed3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:12.529545 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2276acd1-312e-4cd6-976a-f867627a5ed3 service cinder] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:12.539159 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1080/2149] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:57:12 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:57:13.804229 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ea3079-133b-4252-8f6c-685895884c77 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:13.805118 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ea3079-133b-4252-8f6c-685895884c77 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:13.805118 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ea3079-133b-4252-8f6c-685895884c77 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:13.806814 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1070/2150] 10.222.0.52 () {54 vars in 921 bytes} [Sun Sep 25 00:57:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:57:13.822113 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1774a40-bced-4d19-9936-cfd8740f28ce None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:13.822696 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1774a40-bced-4d19-9936-cfd8740f28ce None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:13.823193 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1774a40-bced-4d19-9936-cfd8740f28ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:13.845146 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-d1774a40-bced-4d19-9936-cfd8740f28ce None None] MFA Rules not processed for user `11a80b55dcd444df901c8bf80f2a2c26`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:57:13.846914 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-d1774a40-bced-4d19-9936-cfd8740f28ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:13.935609 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1081/2151] 10.222.0.52 () {58 vars in 1018 bytes} [Sun Sep 25 00:57:13 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:57:13.962477 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3ea3079-133b-4252-8f6c-685895884c77 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:13.964224 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-e3ea3079-133b-4252-8f6c-685895884c77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:13.983960 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3ea3079-133b-4252-8f6c-685895884c77 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:13.997967 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c2485b1-b1ff-4f65-8638-b7da8b8dbf2d service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '11a80b55dcd444df901c8bf80f2a2c26', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:14.001313 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2485b1-b1ff-4f65-8638-b7da8b8dbf2d service placement] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:14.001938 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2485b1-b1ff-4f65-8638-b7da8b8dbf2d service placement] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:14.002376 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2485b1-b1ff-4f65-8638-b7da8b8dbf2d service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:14.014532 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c2485b1-b1ff-4f65-8638-b7da8b8dbf2d service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:14.016689 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c2485b1-b1ff-4f65-8638-b7da8b8dbf2d service placement] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:14.040569 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1071/2152] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:57:13 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 78 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:57:14.999984 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1774a40-bced-4d19-9936-cfd8740f28ce None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:15.001543 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-d1774a40-bced-4d19-9936-cfd8740f28ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:15.040721 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:57:15.078686 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1774a40-bced-4d19-9936-cfd8740f28ce None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:15.083320 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e35dd8-e5b2-46c4-a9bc-6a7f90d545e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:15.084095 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e35dd8-e5b2-46c4-a9bc-6a7f90d545e0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:15.084366 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e35dd8-e5b2-46c4-a9bc-6a7f90d545e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:15.084630 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e35dd8-e5b2-46c4-a9bc-6a7f90d545e0 admin admin] PATH_INFO: `/v3/users/0e9e011d2f014022968bed46bd10c2fb` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:15.085762 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10e35dd8-e5b2-46c4-a9bc-6a7f90d545e0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0e9e011d2f014022968bed46bd10c2fb)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:15.087665 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10e35dd8-e5b2-46c4-a9bc-6a7f90d545e0 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:15.150421 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-10e35dd8-e5b2-46c4-a9bc-6a7f90d545e0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0e9e011d2f014022968bed46bd10c2fb'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:15.150981 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-10e35dd8-e5b2-46c4-a9bc-6a7f90d545e0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0e9e011d2f014022968bed46bd10c2fb'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:15.158967 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-10e35dd8-e5b2-46c4-a9bc-6a7f90d545e0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0e9e011d2f014022968bed46bd10c2fb'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:15.159575 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-10e35dd8-e5b2-46c4-a9bc-6a7f90d545e0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0e9e011d2f014022968bed46bd10c2fb'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:15.167147 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-10e35dd8-e5b2-46c4-a9bc-6a7f90d545e0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0e9e011d2f014022968bed46bd10c2fb'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:15.184878 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1082/2153] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:57:14 2022] DELETE /identity/v3/users/0e9e011d2f014022968bed46bd10c2fb => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:57:15.395008 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c2485b1-b1ff-4f65-8638-b7da8b8dbf2d service placement] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:15.408806 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c2485b1-b1ff-4f65-8638-b7da8b8dbf2d service placement] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:15.492028 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8b7df64-6fde-4015-af85-6f9465d7b208 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:15.492700 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b7df64-6fde-4015-af85-6f9465d7b208 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:15.492972 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b7df64-6fde-4015-af85-6f9465d7b208 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:15.493239 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b7df64-6fde-4015-af85-6f9465d7b208 admin admin] PATH_INFO: `/v3/projects/e463fe4d96b04d6fa3224a417eeeeadc` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:15.494413 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8b7df64-6fde-4015-af85-6f9465d7b208 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e463fe4d96b04d6fa3224a417eeeeadc)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:15.498958 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8b7df64-6fde-4015-af85-6f9465d7b208 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:15.555085 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-b8b7df64-6fde-4015-af85-6f9465d7b208 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e463fe4d96b04d6fa3224a417eeeeadc', 'request_id': 'req-b8b7df64-6fde-4015-af85-6f9465d7b208'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:15.563383 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-b8b7df64-6fde-4015-af85-6f9465d7b208 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e463fe4d96b04d6fa3224a417eeeeadc', 'request_id': 'req-b8b7df64-6fde-4015-af85-6f9465d7b208'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:15.569239 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:57:15.571040 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-b8b7df64-6fde-4015-af85-6f9465d7b208 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e463fe4d96b04d6fa3224a417eeeeadc', 'request_id': 'req-b8b7df64-6fde-4015-af85-6f9465d7b208'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:15.602986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-b8b7df64-6fde-4015-af85-6f9465d7b208 admin admin] The token cache is being invalidate because project e463fe4d96b04d6fa3224a417eeeeadc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81386) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:57:15.603277 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-b8b7df64-6fde-4015-af85-6f9465d7b208 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:15.603941 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-b8b7df64-6fde-4015-af85-6f9465d7b208 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:15.608487 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1072/2154] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:57:15 2022] DELETE /identity/v3/projects/e463fe4d96b04d6fa3224a417eeeeadc => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:57:15.609932 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e35dd8-e5b2-46c4-a9bc-6a7f90d545e0 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:15.611456 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-10e35dd8-e5b2-46c4-a9bc-6a7f90d545e0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:15.629634 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e35dd8-e5b2-46c4-a9bc-6a7f90d545e0 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:15.700430 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6dc9fdc-69fb-41bb-acd0-99348558cfcd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:15.701144 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6dc9fdc-69fb-41bb-acd0-99348558cfcd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:15.701513 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6dc9fdc-69fb-41bb-acd0-99348558cfcd admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:15.701910 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6dc9fdc-69fb-41bb-acd0-99348558cfcd admin admin] PATH_INFO: `/v3/users/ea301e7372b841cba1ebf8c81699c230` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:15.703221 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6dc9fdc-69fb-41bb-acd0-99348558cfcd admin admin] RBAC: Authorizing `identity:delete_user(user_id=ea301e7372b841cba1ebf8c81699c230)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:15.705142 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6dc9fdc-69fb-41bb-acd0-99348558cfcd admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:15.753299 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-a6dc9fdc-69fb-41bb-acd0-99348558cfcd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ea301e7372b841cba1ebf8c81699c230'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:15.753934 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-a6dc9fdc-69fb-41bb-acd0-99348558cfcd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ea301e7372b841cba1ebf8c81699c230'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:15.761899 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-a6dc9fdc-69fb-41bb-acd0-99348558cfcd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ea301e7372b841cba1ebf8c81699c230'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:15.762487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-a6dc9fdc-69fb-41bb-acd0-99348558cfcd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ea301e7372b841cba1ebf8c81699c230'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:15.763749 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=80,keystone:DELETE=18,keystone:INSERT=2 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:57:15.771548 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-a6dc9fdc-69fb-41bb-acd0-99348558cfcd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ea301e7372b841cba1ebf8c81699c230'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:15.789396 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1083/2155] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:57:15 2022] DELETE /identity/v3/users/ea301e7372b841cba1ebf8c81699c230 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:57:16.000016 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8b7df64-6fde-4015-af85-6f9465d7b208 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:16.001556 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-b8b7df64-6fde-4015-af85-6f9465d7b208 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:16.017524 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8b7df64-6fde-4015-af85-6f9465d7b208 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:16.087738 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22712804-a3b9-45b8-96d6-6deffd341118 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:16.088375 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22712804-a3b9-45b8-96d6-6deffd341118 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:16.088650 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22712804-a3b9-45b8-96d6-6deffd341118 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:16.088920 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22712804-a3b9-45b8-96d6-6deffd341118 admin admin] PATH_INFO: `/v3/projects/254d347551d748c3b8356d11a8e309c1` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:16.090069 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22712804-a3b9-45b8-96d6-6deffd341118 admin admin] RBAC: Authorizing `identity:delete_project(project_id=254d347551d748c3b8356d11a8e309c1)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:16.091976 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22712804-a3b9-45b8-96d6-6deffd341118 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:16.145003 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-22712804-a3b9-45b8-96d6-6deffd341118 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '254d347551d748c3b8356d11a8e309c1', 'request_id': 'req-22712804-a3b9-45b8-96d6-6deffd341118'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:16.155853 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-22712804-a3b9-45b8-96d6-6deffd341118 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '254d347551d748c3b8356d11a8e309c1', 'request_id': 'req-22712804-a3b9-45b8-96d6-6deffd341118'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:16.164183 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-22712804-a3b9-45b8-96d6-6deffd341118 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '254d347551d748c3b8356d11a8e309c1', 'request_id': 'req-22712804-a3b9-45b8-96d6-6deffd341118'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:16.177679 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-22712804-a3b9-45b8-96d6-6deffd341118 admin admin] The token cache is being invalidate because project 254d347551d748c3b8356d11a8e309c1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81386) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:57:16.178064 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-22712804-a3b9-45b8-96d6-6deffd341118 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:16.179042 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-22712804-a3b9-45b8-96d6-6deffd341118 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:16.180963 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1073/2156] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:57:15 2022] DELETE /identity/v3/projects/254d347551d748c3b8356d11a8e309c1 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:57:16.188651 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa363b36-1129-4c26-8661-08758f4b9298 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:16.189172 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa363b36-1129-4c26-8661-08758f4b9298 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:16.189312 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa363b36-1129-4c26-8661-08758f4b9298 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:16.219186 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-fa363b36-1129-4c26-8661-08758f4b9298 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:57:16.220389 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-fa363b36-1129-4c26-8661-08758f4b9298 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:16.296277 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1084/2157] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:57:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 00:57:16.304460 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca999c81-b81f-4277-8c5a-3f55fefddfe2 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:16.304736 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca999c81-b81f-4277-8c5a-3f55fefddfe2 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:16.305014 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca999c81-b81f-4277-8c5a-3f55fefddfe2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:16.330055 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-ca999c81-b81f-4277-8c5a-3f55fefddfe2 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:57:16.331379 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-ca999c81-b81f-4277-8c5a-3f55fefddfe2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:16.340493 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1074/2158] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:57:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:57:16.345462 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa363b36-1129-4c26-8661-08758f4b9298 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:16.358199 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa363b36-1129-4c26-8661-08758f4b9298 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:16.363491 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98319c2e-5d85-41a9-ba17-ed3042daf7af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:16.364159 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98319c2e-5d85-41a9-ba17-ed3042daf7af admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:16.364459 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98319c2e-5d85-41a9-ba17-ed3042daf7af admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:16.364758 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98319c2e-5d85-41a9-ba17-ed3042daf7af admin admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:16.365442 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98319c2e-5d85-41a9-ba17-ed3042daf7af admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:57:16.366000 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98319c2e-5d85-41a9-ba17-ed3042daf7af admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:16.367646 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98319c2e-5d85-41a9-ba17-ed3042daf7af admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:16.380427 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1085/2159] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 00:57:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:57:16.385128 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca999c81-b81f-4277-8c5a-3f55fefddfe2 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:16.394706 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca999c81-b81f-4277-8c5a-3f55fefddfe2 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:16.400134 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e26c916-aa65-4a57-a780-b6bc346534d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:16.400742 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e26c916-aa65-4a57-a780-b6bc346534d0 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:16.401018 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e26c916-aa65-4a57-a780-b6bc346534d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:16.401318 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e26c916-aa65-4a57-a780-b6bc346534d0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:16.401979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e26c916-aa65-4a57-a780-b6bc346534d0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:16.403875 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e26c916-aa65-4a57-a780-b6bc346534d0 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:16.446849 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1075/2160] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:57:16 2022] POST /identity/v3/projects => generated 399 bytes in 62 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:57:16.451185 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98319c2e-5d85-41a9-ba17-ed3042daf7af admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:16.462599 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98319c2e-5d85-41a9-ba17-ed3042daf7af admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:16.543247 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bfd9c27-73e1-4059-8147-0996ffdaad8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:16.543863 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfd9c27-73e1-4059-8147-0996ffdaad8a admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:16.544139 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfd9c27-73e1-4059-8147-0996ffdaad8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:16.544408 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfd9c27-73e1-4059-8147-0996ffdaad8a admin admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:16.545039 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bfd9c27-73e1-4059-8147-0996ffdaad8a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:16.546992 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bfd9c27-73e1-4059-8147-0996ffdaad8a admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:16.581833 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1086/2161] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:57:16 2022] POST /identity/v3/users => generated 300 bytes in 128 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:57:16.585872 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e26c916-aa65-4a57-a780-b6bc346534d0 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:16.597142 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e26c916-aa65-4a57-a780-b6bc346534d0 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:16.603236 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72dde00-6a60-4006-9468-58359e8d0c53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:16.603892 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72dde00-6a60-4006-9468-58359e8d0c53 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:16.604178 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72dde00-6a60-4006-9468-58359e8d0c53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:16.604441 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72dde00-6a60-4006-9468-58359e8d0c53 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:16.605088 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f72dde00-6a60-4006-9468-58359e8d0c53 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:16.606750 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f72dde00-6a60-4006-9468-58359e8d0c53 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:16.618664 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1076/2162] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:57:16 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:57:16.623189 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bfd9c27-73e1-4059-8147-0996ffdaad8a admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:16.635342 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bfd9c27-73e1-4059-8147-0996ffdaad8a admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:16.640495 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1bf5610-3a36-4de6-89f8-9eb06c9abf24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:16.641153 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1bf5610-3a36-4de6-89f8-9eb06c9abf24 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:16.641523 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1bf5610-3a36-4de6-89f8-9eb06c9abf24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:16.641832 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1bf5610-3a36-4de6-89f8-9eb06c9abf24 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:16.642525 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1bf5610-3a36-4de6-89f8-9eb06c9abf24 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:16.644224 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1bf5610-3a36-4de6-89f8-9eb06c9abf24 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:16.659205 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1087/2163] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:57:16 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:57:16.664449 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72dde00-6a60-4006-9468-58359e8d0c53 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:16.674988 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72dde00-6a60-4006-9468-58359e8d0c53 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:16.680112 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5cc5ac-36d3-4c8b-bcb6-7af18a79f00e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:16.680642 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5cc5ac-36d3-4c8b-bcb6-7af18a79f00e admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:16.680886 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5cc5ac-36d3-4c8b-bcb6-7af18a79f00e admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:16.681241 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5cc5ac-36d3-4c8b-bcb6-7af18a79f00e admin admin] PATH_INFO: `/v3/projects/27ee0e177ad6403eaa8602e1e0416ddf/users/6db579c3cb40470fadcd8b0dc8ab3a50/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:16.709626 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be5cc5ac-36d3-4c8b-bcb6-7af18a79f00e admin admin] RBAC: Authorizing `identity:create_grant(project_id=27ee0e177ad6403eaa8602e1e0416ddf, user_id=6db579c3cb40470fadcd8b0dc8ab3a50, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:16.712300 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be5cc5ac-36d3-4c8b-bcb6-7af18a79f00e admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:16.729814 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1077/2164] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:57:16 2022] PUT /identity/v3/projects/27ee0e177ad6403eaa8602e1e0416ddf/users/6db579c3cb40470fadcd8b0dc8ab3a50/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:57:16.740743 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dcef00a-1b13-4405-9d8e-707cd5684df6 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:16.741195 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dcef00a-1b13-4405-9d8e-707cd5684df6 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:16.741570 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dcef00a-1b13-4405-9d8e-707cd5684df6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:16.772238 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-3dcef00a-1b13-4405-9d8e-707cd5684df6 None None] MFA Rules not processed for user `6db579c3cb40470fadcd8b0dc8ab3a50`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:57:16.773763 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-3dcef00a-1b13-4405-9d8e-707cd5684df6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:16.803997 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=95,keystone:DELETE=2,keystone:INSERT=3 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:57:16.855842 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1088/2165] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:57:16 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:57:16.868740 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5cc5ac-36d3-4c8b-bcb6-7af18a79f00e admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:16.870515 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-be5cc5ac-36d3-4c8b-bcb6-7af18a79f00e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:16.902927 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:57:16.948617 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5cc5ac-36d3-4c8b-bcb6-7af18a79f00e admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:16.952177 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4751220c-da13-4fc7-982d-3734c39e29ca service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:16.952771 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4751220c-da13-4fc7-982d-3734c39e29ca service cinder] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:16.953040 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4751220c-da13-4fc7-982d-3734c39e29ca service cinder] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:16.953328 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4751220c-da13-4fc7-982d-3734c39e29ca service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:16.962471 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4751220c-da13-4fc7-982d-3734c39e29ca service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:16.963881 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4751220c-da13-4fc7-982d-3734c39e29ca service cinder] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:16.972633 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1078/2166] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:57:16 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:57:24.517391 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dcef00a-1b13-4405-9d8e-707cd5684df6 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:24.518858 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-3dcef00a-1b13-4405-9d8e-707cd5684df6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:24.585854 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dcef00a-1b13-4405-9d8e-707cd5684df6 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:24.590348 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b6d7d32-6d88-4da2-a3bd-793c79ccc6bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:24.590965 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6d7d32-6d88-4da2-a3bd-793c79ccc6bf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:24.591236 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6d7d32-6d88-4da2-a3bd-793c79ccc6bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:24.591499 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6d7d32-6d88-4da2-a3bd-793c79ccc6bf admin admin] PATH_INFO: `/v3/users/71f43ed0cee749b9be1df93f216157b8` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:24.592607 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b6d7d32-6d88-4da2-a3bd-793c79ccc6bf admin admin] RBAC: Authorizing `identity:delete_user(user_id=71f43ed0cee749b9be1df93f216157b8)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:24.594542 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b6d7d32-6d88-4da2-a3bd-793c79ccc6bf admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:24.644642 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-5b6d7d32-6d88-4da2-a3bd-793c79ccc6bf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '71f43ed0cee749b9be1df93f216157b8'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:24.645347 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-5b6d7d32-6d88-4da2-a3bd-793c79ccc6bf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '71f43ed0cee749b9be1df93f216157b8'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:24.652942 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-5b6d7d32-6d88-4da2-a3bd-793c79ccc6bf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '71f43ed0cee749b9be1df93f216157b8'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:24.654038 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-5b6d7d32-6d88-4da2-a3bd-793c79ccc6bf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '71f43ed0cee749b9be1df93f216157b8'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:24.663801 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-5b6d7d32-6d88-4da2-a3bd-793c79ccc6bf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '71f43ed0cee749b9be1df93f216157b8'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:24.689138 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1089/2167] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:57:24 2022] DELETE /identity/v3/users/71f43ed0cee749b9be1df93f216157b8 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:57:24.694160 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9191aeaa-8da6-415a-ae0a-14db385934fd None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:24.694634 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9191aeaa-8da6-415a-ae0a-14db385934fd None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:24.694728 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9191aeaa-8da6-415a-ae0a-14db385934fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:24.716611 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-9191aeaa-8da6-415a-ae0a-14db385934fd None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:57:24.717978 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-9191aeaa-8da6-415a-ae0a-14db385934fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:24.784680 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1079/2168] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:57:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:57:24.795140 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b6d7d32-6d88-4da2-a3bd-793c79ccc6bf admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:24.796606 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-5b6d7d32-6d88-4da2-a3bd-793c79ccc6bf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:24.854264 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:57:24.859736 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b6d7d32-6d88-4da2-a3bd-793c79ccc6bf admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:24.865693 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-744519af-cf2d-4f41-ab0d-8bc985a2e016 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:24.866728 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744519af-cf2d-4f41-ab0d-8bc985a2e016 service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:24.867074 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744519af-cf2d-4f41-ab0d-8bc985a2e016 service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:24.867594 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744519af-cf2d-4f41-ab0d-8bc985a2e016 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:24.880846 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-744519af-cf2d-4f41-ab0d-8bc985a2e016 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:24.884136 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-744519af-cf2d-4f41-ab0d-8bc985a2e016 service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:24.905477 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1090/2169] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:57:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:57:25.254199 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9191aeaa-8da6-415a-ae0a-14db385934fd None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:25.257126 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-9191aeaa-8da6-415a-ae0a-14db385934fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:25.271129 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9191aeaa-8da6-415a-ae0a-14db385934fd None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:25.280377 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7176ba66-9127-41c0-9f68-2a9ab3f40417 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:25.281854 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7176ba66-9127-41c0-9f68-2a9ab3f40417 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:25.282352 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7176ba66-9127-41c0-9f68-2a9ab3f40417 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:25.282597 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7176ba66-9127-41c0-9f68-2a9ab3f40417 admin admin] PATH_INFO: `/v3/users/6db579c3cb40470fadcd8b0dc8ab3a50` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:25.284330 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7176ba66-9127-41c0-9f68-2a9ab3f40417 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6db579c3cb40470fadcd8b0dc8ab3a50)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:25.287461 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7176ba66-9127-41c0-9f68-2a9ab3f40417 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:25.301230 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-744519af-cf2d-4f41-ab0d-8bc985a2e016 service neutron] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:25.303334 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-744519af-cf2d-4f41-ab0d-8bc985a2e016 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:25.318757 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-744519af-cf2d-4f41-ab0d-8bc985a2e016 service neutron] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:25.324213 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e603514-d25f-42a4-beff-7358988af5aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:25.324845 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e603514-d25f-42a4-beff-7358988af5aa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:25.325119 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e603514-d25f-42a4-beff-7358988af5aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:25.325505 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e603514-d25f-42a4-beff-7358988af5aa admin admin] PATH_INFO: `/v3/projects/7bc12e750566459ebac4a268b257f1c7` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:25.326900 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e603514-d25f-42a4-beff-7358988af5aa admin admin] RBAC: Authorizing `identity:delete_project(project_id=7bc12e750566459ebac4a268b257f1c7)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:25.328782 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e603514-d25f-42a4-beff-7358988af5aa admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:25.396465 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-1e603514-d25f-42a4-beff-7358988af5aa admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7bc12e750566459ebac4a268b257f1c7', 'request_id': 'req-1e603514-d25f-42a4-beff-7358988af5aa'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:25.404835 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-1e603514-d25f-42a4-beff-7358988af5aa admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7bc12e750566459ebac4a268b257f1c7', 'request_id': 'req-1e603514-d25f-42a4-beff-7358988af5aa'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:25.405560 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-7176ba66-9127-41c0-9f68-2a9ab3f40417 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6db579c3cb40470fadcd8b0dc8ab3a50'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:25.413240 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-1e603514-d25f-42a4-beff-7358988af5aa admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7bc12e750566459ebac4a268b257f1c7', 'request_id': 'req-1e603514-d25f-42a4-beff-7358988af5aa'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:25.414102 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-7176ba66-9127-41c0-9f68-2a9ab3f40417 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6db579c3cb40470fadcd8b0dc8ab3a50'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:25.414905 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-7176ba66-9127-41c0-9f68-2a9ab3f40417 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6db579c3cb40470fadcd8b0dc8ab3a50'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:25.423723 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-7176ba66-9127-41c0-9f68-2a9ab3f40417 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6db579c3cb40470fadcd8b0dc8ab3a50'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:25.424468 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-7176ba66-9127-41c0-9f68-2a9ab3f40417 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6db579c3cb40470fadcd8b0dc8ab3a50'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:25.427585 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-1e603514-d25f-42a4-beff-7358988af5aa admin admin] The token cache is being invalidate because project 7bc12e750566459ebac4a268b257f1c7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:57:25.427953 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-1e603514-d25f-42a4-beff-7358988af5aa admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:25.428686 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-1e603514-d25f-42a4-beff-7358988af5aa admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:25.430652 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1091/2170] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:57:25 2022] DELETE /identity/v3/projects/7bc12e750566459ebac4a268b257f1c7 => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 00:57:25.435663 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f902ea31-c8d4-40fc-940d-81f11bd6c79f None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:25.436986 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f902ea31-c8d4-40fc-940d-81f11bd6c79f None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:25.437515 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f902ea31-c8d4-40fc-940d-81f11bd6c79f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:25.454797 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1080/2171] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:57:25 2022] DELETE /identity/v3/users/6db579c3cb40470fadcd8b0dc8ab3a50 => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:57:25.460016 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577a901d-6555-47ed-8729-7cb201d91d0f None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:25.460296 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577a901d-6555-47ed-8729-7cb201d91d0f None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:25.460559 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577a901d-6555-47ed-8729-7cb201d91d0f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:25.464768 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-f902ea31-c8d4-40fc-940d-81f11bd6c79f None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:57:25.466403 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-f902ea31-c8d4-40fc-940d-81f11bd6c79f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:25.484559 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-577a901d-6555-47ed-8729-7cb201d91d0f None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:57:25.486619 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-577a901d-6555-47ed-8729-7cb201d91d0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:25.524258 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:57:25.553538 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1092/2172] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:57:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 00:57:25.558421 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fefc110f-c9c9-4ca0-88fe-d5e4387a5f07 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:25.559196 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fefc110f-c9c9-4ca0-88fe-d5e4387a5f07 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:25.559508 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fefc110f-c9c9-4ca0-88fe-d5e4387a5f07 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:25.573469 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1081/2173] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:57:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 00:57:25.584606 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-fefc110f-c9c9-4ca0-88fe-d5e4387a5f07 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:57:25.592241 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-fefc110f-c9c9-4ca0-88fe-d5e4387a5f07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:25.592489 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-577a901d-6555-47ed-8729-7cb201d91d0f None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:25.597772 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1093/2174] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:57:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:57:25.602008 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-577a901d-6555-47ed-8729-7cb201d91d0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:25.603250 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fefc110f-c9c9-4ca0-88fe-d5e4387a5f07 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:25.613136 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fefc110f-c9c9-4ca0-88fe-d5e4387a5f07 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:25.619149 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483356cb-2c78-4c07-a16d-a741d42b931d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:25.619896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483356cb-2c78-4c07-a16d-a741d42b931d admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:25.620265 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483356cb-2c78-4c07-a16d-a741d42b931d admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:25.620866 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483356cb-2c78-4c07-a16d-a741d42b931d admin admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:25.621395 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-483356cb-2c78-4c07-a16d-a741d42b931d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:57:25.621763 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-483356cb-2c78-4c07-a16d-a741d42b931d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:25.623453 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-483356cb-2c78-4c07-a16d-a741d42b931d admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:25.639878 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1094/2175] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 00:57:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:57:25.640717 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483356cb-2c78-4c07-a16d-a741d42b931d admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:25.651060 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483356cb-2c78-4c07-a16d-a741d42b931d admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:25.655887 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f43b850-8d66-4a50-bc07-30d7f774dc58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:25.656574 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f43b850-8d66-4a50-bc07-30d7f774dc58 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:25.656757 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f43b850-8d66-4a50-bc07-30d7f774dc58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:25.657020 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f43b850-8d66-4a50-bc07-30d7f774dc58 admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:25.657683 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f43b850-8d66-4a50-bc07-30d7f774dc58 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:25.659566 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f43b850-8d66-4a50-bc07-30d7f774dc58 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:25.677779 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-577a901d-6555-47ed-8729-7cb201d91d0f None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:25.681864 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae2857b4-1bb9-429b-b33b-f01ceaa1b769 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:25.682483 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2857b4-1bb9-429b-b33b-f01ceaa1b769 service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:25.682764 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2857b4-1bb9-429b-b33b-f01ceaa1b769 service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:25.683085 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2857b4-1bb9-429b-b33b-f01ceaa1b769 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:25.695007 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae2857b4-1bb9-429b-b33b-f01ceaa1b769 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:25.696473 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae2857b4-1bb9-429b-b33b-f01ceaa1b769 service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:25.697037 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1095/2176] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:57:25 2022] POST /identity/v3/projects => generated 359 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:57:25.700851 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f43b850-8d66-4a50-bc07-30d7f774dc58 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:25.707689 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1082/2177] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:57:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:57:25.748832 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:57:25.790746 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f43b850-8d66-4a50-bc07-30d7f774dc58 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:25.795780 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c31bb0-7b72-4b3c-b349-828f96aba51b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:25.796470 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c31bb0-7b72-4b3c-b349-828f96aba51b admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:25.796827 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c31bb0-7b72-4b3c-b349-828f96aba51b admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:25.797165 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c31bb0-7b72-4b3c-b349-828f96aba51b admin admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:25.798510 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c31bb0-7b72-4b3c-b349-828f96aba51b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:25.800043 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c31bb0-7b72-4b3c-b349-828f96aba51b admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:25.836545 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1096/2178] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:57:25 2022] POST /identity/v3/users => generated 280 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 00:57:25.840173 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae2857b4-1bb9-429b-b33b-f01ceaa1b769 service neutron] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:25.853674 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae2857b4-1bb9-429b-b33b-f01ceaa1b769 service neutron] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:25.860403 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb6eb747-ff1c-4634-8a11-afce4bc58816 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:25.861023 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6eb747-ff1c-4634-8a11-afce4bc58816 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:25.861343 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6eb747-ff1c-4634-8a11-afce4bc58816 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:25.861619 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6eb747-ff1c-4634-8a11-afce4bc58816 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:25.862271 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb6eb747-ff1c-4634-8a11-afce4bc58816 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:25.864971 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb6eb747-ff1c-4634-8a11-afce4bc58816 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:25.876622 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1083/2179] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:57:25 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:57:25.882858 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c31bb0-7b72-4b3c-b349-828f96aba51b admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:25.909523 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c31bb0-7b72-4b3c-b349-828f96aba51b admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:25.921396 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-824fcae7-a31d-4401-b1b2-8a4e4864dc81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:25.922294 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824fcae7-a31d-4401-b1b2-8a4e4864dc81 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:25.922660 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824fcae7-a31d-4401-b1b2-8a4e4864dc81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:25.923002 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824fcae7-a31d-4401-b1b2-8a4e4864dc81 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:25.923732 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-824fcae7-a31d-4401-b1b2-8a4e4864dc81 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:25.925486 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-824fcae7-a31d-4401-b1b2-8a4e4864dc81 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:25.945220 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1097/2180] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:57:25 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:57:25.945873 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb6eb747-ff1c-4634-8a11-afce4bc58816 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:25.962235 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb6eb747-ff1c-4634-8a11-afce4bc58816 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:25.968804 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e6aa0b-2a6d-47ed-a1e3-580ebb216280 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:25.969513 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e6aa0b-2a6d-47ed-a1e3-580ebb216280 admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:25.969776 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e6aa0b-2a6d-47ed-a1e3-580ebb216280 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:25.970051 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e6aa0b-2a6d-47ed-a1e3-580ebb216280 admin admin] PATH_INFO: `/v3/projects/fe7866402a0745b6893cfa838864cfe7/users/0b8ca9b3ee14409aa7398c269f2ebdb0/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:25.994815 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e6aa0b-2a6d-47ed-a1e3-580ebb216280 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fe7866402a0745b6893cfa838864cfe7, user_id=0b8ca9b3ee14409aa7398c269f2ebdb0, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:25.997647 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e6aa0b-2a6d-47ed-a1e3-580ebb216280 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:26.022461 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1084/2181] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:57:25 2022] PUT /identity/v3/projects/fe7866402a0745b6893cfa838864cfe7/users/0b8ca9b3ee14409aa7398c269f2ebdb0/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 71 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:57:26.028857 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b19b6b-e411-43c9-b1c7-92e2cd3a5f3a None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:26.033645 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b19b6b-e411-43c9-b1c7-92e2cd3a5f3a None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:26.036628 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b19b6b-e411-43c9-b1c7-92e2cd3a5f3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:26.063892 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e6aa0b-2a6d-47ed-a1e3-580ebb216280 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:26.065920 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-f1e6aa0b-2a6d-47ed-a1e3-580ebb216280 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:26.087959 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-41b19b6b-e411-43c9-b1c7-92e2cd3a5f3a None None] MFA Rules not processed for user `0b8ca9b3ee14409aa7398c269f2ebdb0`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:57:26.089300 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-41b19b6b-e411-43c9-b1c7-92e2cd3a5f3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:26.197156 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1098/2182] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:57:26 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:57:26.201146 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e6aa0b-2a6d-47ed-a1e3-580ebb216280 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:26.206302 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49136849-fe0a-4227-ba0f-5ec4a4cd6534 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:26.207117 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49136849-fe0a-4227-ba0f-5ec4a4cd6534 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:26.208120 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49136849-fe0a-4227-ba0f-5ec4a4cd6534 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:26.208382 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49136849-fe0a-4227-ba0f-5ec4a4cd6534 admin admin] PATH_INFO: `/v3/projects/27ee0e177ad6403eaa8602e1e0416ddf` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:26.209576 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49136849-fe0a-4227-ba0f-5ec4a4cd6534 admin admin] RBAC: Authorizing `identity:delete_project(project_id=27ee0e177ad6403eaa8602e1e0416ddf)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:26.211484 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49136849-fe0a-4227-ba0f-5ec4a4cd6534 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:26.250040 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b19b6b-e411-43c9-b1c7-92e2cd3a5f3a None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:26.252978 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-41b19b6b-e411-43c9-b1c7-92e2cd3a5f3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:26.276330 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:57:26.283220 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-49136849-fe0a-4227-ba0f-5ec4a4cd6534 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '27ee0e177ad6403eaa8602e1e0416ddf', 'request_id': 'req-49136849-fe0a-4227-ba0f-5ec4a4cd6534'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:26.297342 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-49136849-fe0a-4227-ba0f-5ec4a4cd6534 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '27ee0e177ad6403eaa8602e1e0416ddf', 'request_id': 'req-49136849-fe0a-4227-ba0f-5ec4a4cd6534'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:26.305920 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-49136849-fe0a-4227-ba0f-5ec4a4cd6534 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '27ee0e177ad6403eaa8602e1e0416ddf', 'request_id': 'req-49136849-fe0a-4227-ba0f-5ec4a4cd6534'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:26.320915 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-49136849-fe0a-4227-ba0f-5ec4a4cd6534 admin admin] The token cache is being invalidate because project 27ee0e177ad6403eaa8602e1e0416ddf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81386) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:57:26.321799 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-49136849-fe0a-4227-ba0f-5ec4a4cd6534 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:26.323071 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-49136849-fe0a-4227-ba0f-5ec4a4cd6534 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:26.324935 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1085/2183] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:57:26 2022] DELETE /identity/v3/projects/27ee0e177ad6403eaa8602e1e0416ddf => generated 0 bytes in 263 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:57:26.331319 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acbfd9e6-2d28-4cfd-a4ee-b9787ed38352 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:26.331814 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acbfd9e6-2d28-4cfd-a4ee-b9787ed38352 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:26.332328 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acbfd9e6-2d28-4cfd-a4ee-b9787ed38352 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:26.343185 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b19b6b-e411-43c9-b1c7-92e2cd3a5f3a None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:26.368699 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-acbfd9e6-2d28-4cfd-a4ee-b9787ed38352 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:57:26.370126 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-acbfd9e6-2d28-4cfd-a4ee-b9787ed38352 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:26.415126 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4003a5e8-4513-49bc-a97b-56e2e40b46f8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:26.415742 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4003a5e8-4513-49bc-a97b-56e2e40b46f8 service cinder] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:26.416018 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4003a5e8-4513-49bc-a97b-56e2e40b46f8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:26.416387 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4003a5e8-4513-49bc-a97b-56e2e40b46f8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:26.418155 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-4003a5e8-4513-49bc-a97b-56e2e40b46f8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:26.428690 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:57:26.454150 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1086/2184] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:57:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:57:26.463337 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b774365-af42-4cd5-a6ed-a810cc4ee033 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:26.463624 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b774365-af42-4cd5-a6ed-a810cc4ee033 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:26.463911 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b774365-af42-4cd5-a6ed-a810cc4ee033 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:26.484538 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-1b774365-af42-4cd5-a6ed-a810cc4ee033 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:57:26.485873 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-1b774365-af42-4cd5-a6ed-a810cc4ee033 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:26.496079 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1087/2185] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:57:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 00:57:26.499887 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b774365-af42-4cd5-a6ed-a810cc4ee033 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:26.506715 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4003a5e8-4513-49bc-a97b-56e2e40b46f8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:26.510788 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4003a5e8-4513-49bc-a97b-56e2e40b46f8 service cinder] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:26.513450 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b774365-af42-4cd5-a6ed-a810cc4ee033 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:26.517974 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af242e1a-21a4-410b-8bbe-44b298fe47f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:26.518669 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af242e1a-21a4-410b-8bbe-44b298fe47f9 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:26.519035 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af242e1a-21a4-410b-8bbe-44b298fe47f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:26.519375 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af242e1a-21a4-410b-8bbe-44b298fe47f9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:26.520313 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af242e1a-21a4-410b-8bbe-44b298fe47f9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:57:26.520740 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af242e1a-21a4-410b-8bbe-44b298fe47f9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:26.522541 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af242e1a-21a4-410b-8bbe-44b298fe47f9 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:26.523325 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1099/2186] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:57:26 2022] GET /identity/v3/auth/tokens => generated 3236 bytes in 274 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 00:57:26.538700 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4003a5e8-4513-49bc-a97b-56e2e40b46f8 service cinder] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:26.540229 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-4003a5e8-4513-49bc-a97b-56e2e40b46f8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:26.540346 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1088/2187] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 00:57:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:57:26.544424 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af242e1a-21a4-410b-8bbe-44b298fe47f9 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:26.556764 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af242e1a-21a4-410b-8bbe-44b298fe47f9 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:26.562784 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4003a5e8-4513-49bc-a97b-56e2e40b46f8 service cinder] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:26.566576 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41782f91-379b-4996-8bf3-aef36ecf22ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:26.568245 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41782f91-379b-4996-8bf3-aef36ecf22ec admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:26.568590 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41782f91-379b-4996-8bf3-aef36ecf22ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:26.568696 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bcca758-1d1f-480a-9d18-c56ea866728e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:26.568969 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41782f91-379b-4996-8bf3-aef36ecf22ec admin admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:26.569337 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bcca758-1d1f-480a-9d18-c56ea866728e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:26.569643 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bcca758-1d1f-480a-9d18-c56ea866728e admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:26.569835 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41782f91-379b-4996-8bf3-aef36ecf22ec admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:26.569956 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bcca758-1d1f-480a-9d18-c56ea866728e admin admin] PATH_INFO: `/v3/users/0b8ca9b3ee14409aa7398c269f2ebdb0` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:26.571145 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bcca758-1d1f-480a-9d18-c56ea866728e admin admin] RBAC: Authorizing `identity:delete_user(user_id=0b8ca9b3ee14409aa7398c269f2ebdb0)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:26.571963 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41782f91-379b-4996-8bf3-aef36ecf22ec admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:26.573227 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bcca758-1d1f-480a-9d18-c56ea866728e admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:26.612778 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1089/2188] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:57:26 2022] POST /identity/v3/projects => generated 379 bytes in 68 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 00:57:26.617116 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41782f91-379b-4996-8bf3-aef36ecf22ec admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:26.638167 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-2bcca758-1d1f-480a-9d18-c56ea866728e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0b8ca9b3ee14409aa7398c269f2ebdb0'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:26.638995 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-2bcca758-1d1f-480a-9d18-c56ea866728e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0b8ca9b3ee14409aa7398c269f2ebdb0'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:26.651737 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-2bcca758-1d1f-480a-9d18-c56ea866728e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0b8ca9b3ee14409aa7398c269f2ebdb0'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:26.652546 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-2bcca758-1d1f-480a-9d18-c56ea866728e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0b8ca9b3ee14409aa7398c269f2ebdb0'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:26.665132 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-2bcca758-1d1f-480a-9d18-c56ea866728e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0b8ca9b3ee14409aa7398c269f2ebdb0'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:26.700543 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1100/2189] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:57:26 2022] DELETE /identity/v3/users/0b8ca9b3ee14409aa7398c269f2ebdb0 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:57:26.705011 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33d878d-0a52-4791-b169-9365f954904c None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:26.705333 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33d878d-0a52-4791-b169-9365f954904c None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:26.705678 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33d878d-0a52-4791-b169-9365f954904c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:26.708898 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41782f91-379b-4996-8bf3-aef36ecf22ec admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:26.734677 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-a33d878d-0a52-4791-b169-9365f954904c None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:57:26.736063 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-a33d878d-0a52-4791-b169-9365f954904c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:26.755017 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=94,keystone:DELETE=4,keystone:INSERT=2 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:57:26.805915 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a4c0f0-7bfa-4ada-b3f1-61f681a6f857 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:26.806408 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a4c0f0-7bfa-4ada-b3f1-61f681a6f857 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:26.806693 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a4c0f0-7bfa-4ada-b3f1-61f681a6f857 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:26.806950 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a4c0f0-7bfa-4ada-b3f1-61f681a6f857 admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:26.807595 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3a4c0f0-7bfa-4ada-b3f1-61f681a6f857 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:26.809771 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3a4c0f0-7bfa-4ada-b3f1-61f681a6f857 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:26.834472 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1101/2190] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:57:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:57:26.837929 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1090/2191] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:57:26 2022] POST /identity/v3/users => generated 290 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 00:57:26.842470 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a4c0f0-7bfa-4ada-b3f1-61f681a6f857 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:26.845189 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a33d878d-0a52-4791-b169-9365f954904c None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:26.846952 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-d3a4c0f0-7bfa-4ada-b3f1-61f681a6f857 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:26.857710 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-a33d878d-0a52-4791-b169-9365f954904c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:26.884658 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a4c0f0-7bfa-4ada-b3f1-61f681a6f857 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:26.891805 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a912c54-dd18-41bf-a24c-5c20b3e50106 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:26.892865 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a912c54-dd18-41bf-a24c-5c20b3e50106 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:26.893620 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a912c54-dd18-41bf-a24c-5c20b3e50106 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:26.894101 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a912c54-dd18-41bf-a24c-5c20b3e50106 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:26.896439 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a912c54-dd18-41bf-a24c-5c20b3e50106 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:26.899843 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a912c54-dd18-41bf-a24c-5c20b3e50106 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:26.919024 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1091/2192] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:57:26 2022] GET /identity/v3/roles => generated 1420 bytes in 77 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:57:26.924461 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a912c54-dd18-41bf-a24c-5c20b3e50106 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:26.927554 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:57:26.942856 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a912c54-dd18-41bf-a24c-5c20b3e50106 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:26.949439 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec787498-6770-4f57-8c98-49c24fd0e1c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:26.949439 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec787498-6770-4f57-8c98-49c24fd0e1c1 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:26.949439 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec787498-6770-4f57-8c98-49c24fd0e1c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:26.949818 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec787498-6770-4f57-8c98-49c24fd0e1c1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:26.950355 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec787498-6770-4f57-8c98-49c24fd0e1c1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:26.952932 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec787498-6770-4f57-8c98-49c24fd0e1c1 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:26.965949 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1092/2193] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:57:26 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:57:26.970780 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a33d878d-0a52-4791-b169-9365f954904c None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:26.972044 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec787498-6770-4f57-8c98-49c24fd0e1c1 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:26.975579 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85125da5-504e-47ae-9e8a-791bbf073e7b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:26.976279 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85125da5-504e-47ae-9e8a-791bbf073e7b service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:26.976638 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85125da5-504e-47ae-9e8a-791bbf073e7b service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:26.976911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85125da5-504e-47ae-9e8a-791bbf073e7b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:26.987836 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85125da5-504e-47ae-9e8a-791bbf073e7b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:26.990051 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec787498-6770-4f57-8c98-49c24fd0e1c1 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:26.990415 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85125da5-504e-47ae-9e8a-791bbf073e7b service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:26.997920 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9515e1e1-aafc-4f1e-8a65-0150e96ac40c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:26.998606 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9515e1e1-aafc-4f1e-8a65-0150e96ac40c admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:26.998878 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9515e1e1-aafc-4f1e-8a65-0150e96ac40c admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:26.999142 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9515e1e1-aafc-4f1e-8a65-0150e96ac40c admin admin] PATH_INFO: `/v3/projects/46f19fd9ad1e4880961d1eee75883687/users/c19b10ebc31244dbb2b1539d94ac281d/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:27.001383 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1102/2194] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:57:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 157 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:57:27.036979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9515e1e1-aafc-4f1e-8a65-0150e96ac40c admin admin] RBAC: Authorizing `identity:create_grant(project_id=46f19fd9ad1e4880961d1eee75883687, user_id=c19b10ebc31244dbb2b1539d94ac281d, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:27.043979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9515e1e1-aafc-4f1e-8a65-0150e96ac40c admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:27.077927 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1093/2195] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:57:26 2022] PUT /identity/v3/projects/46f19fd9ad1e4880961d1eee75883687/users/c19b10ebc31244dbb2b1539d94ac281d/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 00:57:27.083102 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbfd85a-c93e-4891-94d9-dd121d422798 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:27.083320 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbfd85a-c93e-4891-94d9-dd121d422798 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:27.083948 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbfd85a-c93e-4891-94d9-dd121d422798 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:27.107152 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-7fbfd85a-c93e-4891-94d9-dd121d422798 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:57:27.108317 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-7fbfd85a-c93e-4891-94d9-dd121d422798 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:27.233550 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1103/2196] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:57:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:57:27.253618 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9515e1e1-aafc-4f1e-8a65-0150e96ac40c admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:27.351863 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9515e1e1-aafc-4f1e-8a65-0150e96ac40c admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:27.371316 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92d9748-0a27-4713-8411-bbc161dcee63 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:27.372054 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92d9748-0a27-4713-8411-bbc161dcee63 service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:27.372331 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92d9748-0a27-4713-8411-bbc161dcee63 service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:27.373374 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92d9748-0a27-4713-8411-bbc161dcee63 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:27.384682 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a92d9748-0a27-4713-8411-bbc161dcee63 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:27.386229 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a92d9748-0a27-4713-8411-bbc161dcee63 service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:27.397912 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1094/2197] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:57:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:57:27.438096 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fbfd85a-c93e-4891-94d9-dd121d422798 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:27.439785 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-7fbfd85a-c93e-4891-94d9-dd121d422798 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:27.452808 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fbfd85a-c93e-4891-94d9-dd121d422798 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:27.458247 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e672b1e6-1ef5-4e09-b6d4-f2e2958aff8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:27.459037 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e672b1e6-1ef5-4e09-b6d4-f2e2958aff8f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:27.459447 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e672b1e6-1ef5-4e09-b6d4-f2e2958aff8f admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:27.459823 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e672b1e6-1ef5-4e09-b6d4-f2e2958aff8f admin admin] PATH_INFO: `/v3/projects/fe7866402a0745b6893cfa838864cfe7` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:27.461332 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e672b1e6-1ef5-4e09-b6d4-f2e2958aff8f admin admin] RBAC: Authorizing `identity:delete_project(project_id=fe7866402a0745b6893cfa838864cfe7)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:27.464757 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e672b1e6-1ef5-4e09-b6d4-f2e2958aff8f admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:27.550457 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-e672b1e6-1ef5-4e09-b6d4-f2e2958aff8f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fe7866402a0745b6893cfa838864cfe7', 'request_id': 'req-e672b1e6-1ef5-4e09-b6d4-f2e2958aff8f'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:27.560898 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-e672b1e6-1ef5-4e09-b6d4-f2e2958aff8f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fe7866402a0745b6893cfa838864cfe7', 'request_id': 'req-e672b1e6-1ef5-4e09-b6d4-f2e2958aff8f'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:27.568640 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-e672b1e6-1ef5-4e09-b6d4-f2e2958aff8f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fe7866402a0745b6893cfa838864cfe7', 'request_id': 'req-e672b1e6-1ef5-4e09-b6d4-f2e2958aff8f'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:27.582565 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-e672b1e6-1ef5-4e09-b6d4-f2e2958aff8f admin admin] The token cache is being invalidate because project fe7866402a0745b6893cfa838864cfe7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:57:27.582944 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-e672b1e6-1ef5-4e09-b6d4-f2e2958aff8f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:27.585325 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-e672b1e6-1ef5-4e09-b6d4-f2e2958aff8f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:27.587644 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1104/2198] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:57:27 2022] DELETE /identity/v3/projects/fe7866402a0745b6893cfa838864cfe7 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:57:27.593797 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e2922a-1136-44dc-9e74-47766f7d9dba None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:27.594094 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e2922a-1136-44dc-9e74-47766f7d9dba None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:27.594398 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e2922a-1136-44dc-9e74-47766f7d9dba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:27.630428 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-66e2922a-1136-44dc-9e74-47766f7d9dba None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:57:27.631685 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-66e2922a-1136-44dc-9e74-47766f7d9dba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:27.719024 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1095/2199] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:57:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:57:27.729513 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c03d96-3285-4b8b-8548-2a1a310a5cf2 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:27.729795 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c03d96-3285-4b8b-8548-2a1a310a5cf2 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:27.730066 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c03d96-3285-4b8b-8548-2a1a310a5cf2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:27.765433 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-51c03d96-3285-4b8b-8548-2a1a310a5cf2 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:57:27.767056 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-51c03d96-3285-4b8b-8548-2a1a310a5cf2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:27.798824 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1105/2200] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:57:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 71 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 00:57:27.805649 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e2922a-1136-44dc-9e74-47766f7d9dba None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:27.820338 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e2922a-1136-44dc-9e74-47766f7d9dba None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:27.831209 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449683ee-4eba-4bbc-b3ce-f35bdb64a48e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:27.831938 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449683ee-4eba-4bbc-b3ce-f35bdb64a48e admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:27.832468 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449683ee-4eba-4bbc-b3ce-f35bdb64a48e admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:27.833085 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449683ee-4eba-4bbc-b3ce-f35bdb64a48e admin admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:27.835981 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449683ee-4eba-4bbc-b3ce-f35bdb64a48e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:57:27.837745 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449683ee-4eba-4bbc-b3ce-f35bdb64a48e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:27.839597 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449683ee-4eba-4bbc-b3ce-f35bdb64a48e admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:27.873630 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1096/2201] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 00:57:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 70 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:57:27.878535 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c03d96-3285-4b8b-8548-2a1a310a5cf2 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:27.894278 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c03d96-3285-4b8b-8548-2a1a310a5cf2 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:27.907248 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b0b95e-6ac9-4d60-a726-801fa5dc811f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:27.908253 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b0b95e-6ac9-4d60-a726-801fa5dc811f admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:27.908598 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b0b95e-6ac9-4d60-a726-801fa5dc811f admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:27.908938 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b0b95e-6ac9-4d60-a726-801fa5dc811f admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:27.909738 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3b0b95e-6ac9-4d60-a726-801fa5dc811f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:27.912000 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3b0b95e-6ac9-4d60-a726-801fa5dc811f admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:27.958946 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1106/2202] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:57:27 2022] POST /identity/v3/projects => generated 375 bytes in 81 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:57:27.962952 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449683ee-4eba-4bbc-b3ce-f35bdb64a48e admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:27.979940 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449683ee-4eba-4bbc-b3ce-f35bdb64a48e admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:27.992845 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:57:28.130214 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00951822-5074-4640-a969-5a9fe22e5607 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:28.130963 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00951822-5074-4640-a969-5a9fe22e5607 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:28.131245 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00951822-5074-4640-a969-5a9fe22e5607 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:28.131511 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00951822-5074-4640-a969-5a9fe22e5607 admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:28.132148 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00951822-5074-4640-a969-5a9fe22e5607 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:28.141361 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00951822-5074-4640-a969-5a9fe22e5607 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:28.188083 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1097/2203] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:57:27 2022] POST /identity/v3/users => generated 288 bytes in 223 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:57:28.189894 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b0b95e-6ac9-4d60-a726-801fa5dc811f admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:28.203600 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b0b95e-6ac9-4d60-a726-801fa5dc811f admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:28.212536 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be866713-8b3c-4e97-89a1-920ab060ce08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:28.213080 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be866713-8b3c-4e97-89a1-920ab060ce08 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:28.213507 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be866713-8b3c-4e97-89a1-920ab060ce08 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:28.213689 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be866713-8b3c-4e97-89a1-920ab060ce08 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:28.214450 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be866713-8b3c-4e97-89a1-920ab060ce08 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:28.218252 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be866713-8b3c-4e97-89a1-920ab060ce08 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:28.238053 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1107/2204] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:57:28 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:57:28.244560 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00951822-5074-4640-a969-5a9fe22e5607 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:28.262919 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00951822-5074-4640-a969-5a9fe22e5607 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:28.279522 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a7c4c4-13b9-4b75-a56b-54ccc761a3d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:28.280793 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a7c4c4-13b9-4b75-a56b-54ccc761a3d3 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:28.281073 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a7c4c4-13b9-4b75-a56b-54ccc761a3d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:28.281377 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a7c4c4-13b9-4b75-a56b-54ccc761a3d3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:28.282464 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34a7c4c4-13b9-4b75-a56b-54ccc761a3d3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:28.284324 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34a7c4c4-13b9-4b75-a56b-54ccc761a3d3 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:28.321438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1098/2205] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:57:28 2022] GET /identity/v3/roles => generated 1420 bytes in 77 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:57:28.321663 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be866713-8b3c-4e97-89a1-920ab060ce08 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:28.338140 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be866713-8b3c-4e97-89a1-920ab060ce08 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:28.357489 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d67a5ba1-5325-4894-8c36-e2f42bfbc730 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:28.358250 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67a5ba1-5325-4894-8c36-e2f42bfbc730 admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:28.358456 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67a5ba1-5325-4894-8c36-e2f42bfbc730 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:28.358714 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67a5ba1-5325-4894-8c36-e2f42bfbc730 admin admin] PATH_INFO: `/v3/projects/f8f3e47fa0e04e77a29f2a3ef7566c80/users/a9af3af9452b474d887ab282ef684640/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:28.369492 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:57:28.400150 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d67a5ba1-5325-4894-8c36-e2f42bfbc730 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f8f3e47fa0e04e77a29f2a3ef7566c80, user_id=a9af3af9452b474d887ab282ef684640, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:28.405070 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d67a5ba1-5325-4894-8c36-e2f42bfbc730 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:28.449602 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1108/2206] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:57:28 2022] PUT /identity/v3/projects/f8f3e47fa0e04e77a29f2a3ef7566c80/users/a9af3af9452b474d887ab282ef684640/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 00:57:28.461951 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d538def8-d2a6-46b3-9acb-08881062db67 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:28.462227 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d538def8-d2a6-46b3-9acb-08881062db67 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:28.462499 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d538def8-d2a6-46b3-9acb-08881062db67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:28.545972 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-d538def8-d2a6-46b3-9acb-08881062db67 None None] MFA Rules not processed for user `a9af3af9452b474d887ab282ef684640`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:57:28.549182 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-d538def8-d2a6-46b3-9acb-08881062db67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:28.700395 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1099/2207] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:57:28 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 240 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:57:28.712010 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d67a5ba1-5325-4894-8c36-e2f42bfbc730 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:28.715030 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-d67a5ba1-5325-4894-8c36-e2f42bfbc730 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:28.841698 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d67a5ba1-5325-4894-8c36-e2f42bfbc730 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:28.847241 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cef7c89-0088-481d-bb93-6136038445f4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:28.848890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cef7c89-0088-481d-bb93-6136038445f4 service cinder] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:28.848890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cef7c89-0088-481d-bb93-6136038445f4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:28.849347 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cef7c89-0088-481d-bb93-6136038445f4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:28.864162 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cef7c89-0088-481d-bb93-6136038445f4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:28.869092 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cef7c89-0088-481d-bb93-6136038445f4 service cinder] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:28.892205 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1109/2208] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:57:28 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 180 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:57:29.576248 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b994c34c-22ea-4589-8457-c8cea34638ca None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:29.576867 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b994c34c-22ea-4589-8457-c8cea34638ca None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:29.576983 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b994c34c-22ea-4589-8457-c8cea34638ca None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:29.579548 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1100/2209] 10.222.0.52 () {54 vars in 921 bytes} [Sun Sep 25 00:57:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:57:29.590510 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4364c8b5-f9de-40ed-9915-db0a29d2000b None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:29.590798 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4364c8b5-f9de-40ed-9915-db0a29d2000b None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:29.591072 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4364c8b5-f9de-40ed-9915-db0a29d2000b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:29.615935 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-4364c8b5-f9de-40ed-9915-db0a29d2000b None None] MFA Rules not processed for user `11a80b55dcd444df901c8bf80f2a2c26`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:57:29.617180 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-4364c8b5-f9de-40ed-9915-db0a29d2000b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:29.638712 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b994c34c-22ea-4589-8457-c8cea34638ca None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:29.640876 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-b994c34c-22ea-4589-8457-c8cea34638ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:29.744846 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1110/2210] 10.222.0.52 () {58 vars in 1018 bytes} [Sun Sep 25 00:57:29 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:57:29.758684 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4364c8b5-f9de-40ed-9915-db0a29d2000b None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:29.760397 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-4364c8b5-f9de-40ed-9915-db0a29d2000b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:29.770579 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b994c34c-22ea-4589-8457-c8cea34638ca None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:29.774351 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4364c8b5-f9de-40ed-9915-db0a29d2000b None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:29.776513 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c2ceb29-3090-4b45-8b3a-b19c131837dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:29.777606 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2ceb29-3090-4b45-8b3a-b19c131837dc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:29.778094 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2ceb29-3090-4b45-8b3a-b19c131837dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:29.778825 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2ceb29-3090-4b45-8b3a-b19c131837dc admin admin] PATH_INFO: `/v3/users/f6e6706abaea4d0797af19fdb0331ead` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:29.780713 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c2ceb29-3090-4b45-8b3a-b19c131837dc admin admin] RBAC: Authorizing `identity:delete_user(user_id=f6e6706abaea4d0797af19fdb0331ead)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:29.783543 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c2ceb29-3090-4b45-8b3a-b19c131837dc admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:29.784956 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1834249d-2a17-4b29-862c-66a70451fc39 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '11a80b55dcd444df901c8bf80f2a2c26', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:29.786159 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1834249d-2a17-4b29-862c-66a70451fc39 service placement] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:29.786618 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1834249d-2a17-4b29-862c-66a70451fc39 service placement] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:29.787047 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1834249d-2a17-4b29-862c-66a70451fc39 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:29.803837 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1834249d-2a17-4b29-862c-66a70451fc39 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:29.805752 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1834249d-2a17-4b29-862c-66a70451fc39 service placement] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:29.821363 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=94,keystone:INSERT=3,keystone:DELETE=3 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:57:29.830881 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1111/2211] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:57:29 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 73 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 00:57:29.893966 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-6c2ceb29-3090-4b45-8b3a-b19c131837dc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f6e6706abaea4d0797af19fdb0331ead'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:29.906247 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-6c2ceb29-3090-4b45-8b3a-b19c131837dc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f6e6706abaea4d0797af19fdb0331ead'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:29.907246 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-6c2ceb29-3090-4b45-8b3a-b19c131837dc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f6e6706abaea4d0797af19fdb0331ead'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:29.919246 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-6c2ceb29-3090-4b45-8b3a-b19c131837dc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f6e6706abaea4d0797af19fdb0331ead'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:29.920157 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-6c2ceb29-3090-4b45-8b3a-b19c131837dc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f6e6706abaea4d0797af19fdb0331ead'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:29.943844 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1101/2212] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:57:29 2022] DELETE /identity/v3/users/f6e6706abaea4d0797af19fdb0331ead => generated 0 bytes in 306 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 00:57:30.206131 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1834249d-2a17-4b29-862c-66a70451fc39 service placement] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:30.237446 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1834249d-2a17-4b29-862c-66a70451fc39 service placement] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:30.316387 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f3f2bb-b4e7-4f35-8e49-db3fd0cf9e7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:30.317615 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f3f2bb-b4e7-4f35-8e49-db3fd0cf9e7f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:30.317885 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f3f2bb-b4e7-4f35-8e49-db3fd0cf9e7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:30.318159 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f3f2bb-b4e7-4f35-8e49-db3fd0cf9e7f admin admin] PATH_INFO: `/v3/projects/5ca92ea462d24d2d8632e5c52a3dfe63` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:30.319313 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5f3f2bb-b4e7-4f35-8e49-db3fd0cf9e7f admin admin] RBAC: Authorizing `identity:delete_project(project_id=5ca92ea462d24d2d8632e5c52a3dfe63)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:30.321372 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5f3f2bb-b4e7-4f35-8e49-db3fd0cf9e7f admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:30.342472 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:57:30.394303 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-d5f3f2bb-b4e7-4f35-8e49-db3fd0cf9e7f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5ca92ea462d24d2d8632e5c52a3dfe63', 'request_id': 'req-d5f3f2bb-b4e7-4f35-8e49-db3fd0cf9e7f'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:30.402253 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-d5f3f2bb-b4e7-4f35-8e49-db3fd0cf9e7f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5ca92ea462d24d2d8632e5c52a3dfe63', 'request_id': 'req-d5f3f2bb-b4e7-4f35-8e49-db3fd0cf9e7f'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:30.408352 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-d5f3f2bb-b4e7-4f35-8e49-db3fd0cf9e7f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5ca92ea462d24d2d8632e5c52a3dfe63', 'request_id': 'req-d5f3f2bb-b4e7-4f35-8e49-db3fd0cf9e7f'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:30.422217 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-d5f3f2bb-b4e7-4f35-8e49-db3fd0cf9e7f admin admin] The token cache is being invalidate because project 5ca92ea462d24d2d8632e5c52a3dfe63 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:57:30.422500 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-d5f3f2bb-b4e7-4f35-8e49-db3fd0cf9e7f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:30.423152 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-d5f3f2bb-b4e7-4f35-8e49-db3fd0cf9e7f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:30.425298 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1112/2213] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:57:30 2022] DELETE /identity/v3/projects/5ca92ea462d24d2d8632e5c52a3dfe63 => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:57:30.430528 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e849676-59fa-49b0-b828-3275cca92e14 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:30.430875 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e849676-59fa-49b0-b828-3275cca92e14 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:30.431147 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e849676-59fa-49b0-b828-3275cca92e14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:30.483927 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-2e849676-59fa-49b0-b828-3275cca92e14 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:57:30.484802 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-2e849676-59fa-49b0-b828-3275cca92e14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:30.621521 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1102/2214] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:57:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:57:30.629598 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88af1965-594f-440b-89ec-1403fc2c4315 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:30.630706 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88af1965-594f-440b-89ec-1403fc2c4315 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:30.631487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88af1965-594f-440b-89ec-1403fc2c4315 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:30.672453 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-88af1965-594f-440b-89ec-1403fc2c4315 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:57:30.674247 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-88af1965-594f-440b-89ec-1403fc2c4315 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:30.685434 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1113/2215] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:57:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 57 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:57:30.689500 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e849676-59fa-49b0-b828-3275cca92e14 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:30.701295 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e849676-59fa-49b0-b828-3275cca92e14 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:30.706284 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d59231e-bd86-4182-9058-9df71a52cda2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:30.708750 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d59231e-bd86-4182-9058-9df71a52cda2 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:30.709020 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d59231e-bd86-4182-9058-9df71a52cda2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:30.709316 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d59231e-bd86-4182-9058-9df71a52cda2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:30.710104 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d59231e-bd86-4182-9058-9df71a52cda2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:57:30.710406 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d59231e-bd86-4182-9058-9df71a52cda2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:30.712030 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d59231e-bd86-4182-9058-9df71a52cda2 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:30.726466 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1103/2216] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 00:57:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:57:30.730927 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88af1965-594f-440b-89ec-1403fc2c4315 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:30.743461 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88af1965-594f-440b-89ec-1403fc2c4315 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:30.750932 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd84ba3-59dd-45cd-bed0-7b652ae2e281 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:30.752339 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd84ba3-59dd-45cd-bed0-7b652ae2e281 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:30.752840 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd84ba3-59dd-45cd-bed0-7b652ae2e281 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:30.753515 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd84ba3-59dd-45cd-bed0-7b652ae2e281 admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:30.755616 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bd84ba3-59dd-45cd-bed0-7b652ae2e281 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:30.760276 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bd84ba3-59dd-45cd-bed0-7b652ae2e281 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:30.812178 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1114/2217] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:57:30 2022] POST /identity/v3/projects => generated 371 bytes in 81 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:57:30.813242 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d59231e-bd86-4182-9058-9df71a52cda2 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:30.830517 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d59231e-bd86-4182-9058-9df71a52cda2 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:30.927047 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-530df22d-abf8-40ec-869d-3ea72af65286 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:30.927931 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-530df22d-abf8-40ec-869d-3ea72af65286 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:30.928286 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-530df22d-abf8-40ec-869d-3ea72af65286 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:30.928462 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-530df22d-abf8-40ec-869d-3ea72af65286 admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:30.929098 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-530df22d-abf8-40ec-869d-3ea72af65286 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:30.931173 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-530df22d-abf8-40ec-869d-3ea72af65286 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:30.964966 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1104/2218] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:57:30 2022] POST /identity/v3/users => generated 286 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:57:30.969829 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd84ba3-59dd-45cd-bed0-7b652ae2e281 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:30.986899 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd84ba3-59dd-45cd-bed0-7b652ae2e281 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:30.993731 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4288b12d-86de-4a6f-95e3-c86c9718d77b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:30.994587 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4288b12d-86de-4a6f-95e3-c86c9718d77b admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:30.994870 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4288b12d-86de-4a6f-95e3-c86c9718d77b admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:30.995142 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4288b12d-86de-4a6f-95e3-c86c9718d77b admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:30.995832 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4288b12d-86de-4a6f-95e3-c86c9718d77b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:30.997635 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4288b12d-86de-4a6f-95e3-c86c9718d77b admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:31.017336 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1115/2219] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:57:30 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:57:31.018815 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-530df22d-abf8-40ec-869d-3ea72af65286 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:31.031435 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-530df22d-abf8-40ec-869d-3ea72af65286 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:31.037752 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a960642b-32b6-42db-85ec-8f8ee672ff30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:31.038605 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a960642b-32b6-42db-85ec-8f8ee672ff30 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:31.039041 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a960642b-32b6-42db-85ec-8f8ee672ff30 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:31.039447 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a960642b-32b6-42db-85ec-8f8ee672ff30 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:31.040285 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a960642b-32b6-42db-85ec-8f8ee672ff30 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:31.042114 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a960642b-32b6-42db-85ec-8f8ee672ff30 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:31.056447 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1105/2220] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:57:31 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:57:31.062022 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4288b12d-86de-4a6f-95e3-c86c9718d77b admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:31.084133 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4288b12d-86de-4a6f-95e3-c86c9718d77b admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:31.098617 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-067b4bda-c8ae-491e-857f-aeadaee21578 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:31.100737 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-067b4bda-c8ae-491e-857f-aeadaee21578 admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:31.100737 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-067b4bda-c8ae-491e-857f-aeadaee21578 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:31.100984 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-067b4bda-c8ae-491e-857f-aeadaee21578 admin admin] PATH_INFO: `/v3/projects/3880cc1b631a4426adc5cae65932049d/users/b46132eeb4104edf88474e9f80e2405d/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:31.144470 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-067b4bda-c8ae-491e-857f-aeadaee21578 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3880cc1b631a4426adc5cae65932049d, user_id=b46132eeb4104edf88474e9f80e2405d, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:31.152396 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-067b4bda-c8ae-491e-857f-aeadaee21578 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:31.179509 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1116/2221] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:57:31 2022] PUT /identity/v3/projects/3880cc1b631a4426adc5cae65932049d/users/b46132eeb4104edf88474e9f80e2405d/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 00:57:31.191588 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e22be7a-387a-4b0d-aec2-de6acdb2aecf None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:31.191983 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e22be7a-387a-4b0d-aec2-de6acdb2aecf None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:31.192462 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e22be7a-387a-4b0d-aec2-de6acdb2aecf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:31.206611 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:DELETE=7,keystone:SELECT=89,keystone:INSERT=4 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:57:31.282560 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-9e22be7a-387a-4b0d-aec2-de6acdb2aecf None None] MFA Rules not processed for user `b46132eeb4104edf88474e9f80e2405d`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:57:31.288652 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-9e22be7a-387a-4b0d-aec2-de6acdb2aecf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:31.449473 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1106/2222] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:57:31 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 261 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:57:31.450157 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-067b4bda-c8ae-491e-857f-aeadaee21578 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:31.550684 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-067b4bda-c8ae-491e-857f-aeadaee21578 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:31.557388 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f543b16a-40b0-4ebe-9edd-80886b2adc5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:31.558260 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f543b16a-40b0-4ebe-9edd-80886b2adc5b admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:31.558719 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f543b16a-40b0-4ebe-9edd-80886b2adc5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:31.559165 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f543b16a-40b0-4ebe-9edd-80886b2adc5b admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:31.560265 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f543b16a-40b0-4ebe-9edd-80886b2adc5b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:31.563017 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f543b16a-40b0-4ebe-9edd-80886b2adc5b admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:31.618719 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1117/2223] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:57:31 2022] POST /identity/v3/projects => generated 371 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:57:31.622493 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e22be7a-387a-4b0d-aec2-de6acdb2aecf None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:31.638313 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e22be7a-387a-4b0d-aec2-de6acdb2aecf None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:31.804283 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a361379c-6a54-40f7-8afa-0f4129c6a943 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:31.805108 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a361379c-6a54-40f7-8afa-0f4129c6a943 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:31.805516 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a361379c-6a54-40f7-8afa-0f4129c6a943 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:31.805794 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a361379c-6a54-40f7-8afa-0f4129c6a943 admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:31.806451 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a361379c-6a54-40f7-8afa-0f4129c6a943 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:31.808380 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a361379c-6a54-40f7-8afa-0f4129c6a943 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:31.836285 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1107/2224] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:57:31 2022] POST /identity/v3/users => generated 292 bytes in 212 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 00:57:31.837634 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f543b16a-40b0-4ebe-9edd-80886b2adc5b admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:31.849930 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f543b16a-40b0-4ebe-9edd-80886b2adc5b admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:31.857813 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e1b302-e322-4022-8816-902d9bc09851 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:31.858436 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e1b302-e322-4022-8816-902d9bc09851 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:31.858710 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e1b302-e322-4022-8816-902d9bc09851 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:31.858976 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e1b302-e322-4022-8816-902d9bc09851 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:31.859626 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55e1b302-e322-4022-8816-902d9bc09851 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:31.861216 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55e1b302-e322-4022-8816-902d9bc09851 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:31.877179 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1118/2225] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:57:31 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:57:31.877595 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a361379c-6a54-40f7-8afa-0f4129c6a943 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:31.889406 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a361379c-6a54-40f7-8afa-0f4129c6a943 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:31.894260 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-207adba1-6300-48d8-af59-88ebff7083f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:31.896293 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-207adba1-6300-48d8-af59-88ebff7083f6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:31.896642 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-207adba1-6300-48d8-af59-88ebff7083f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:31.896983 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-207adba1-6300-48d8-af59-88ebff7083f6 admin admin] PATH_INFO: `/v3/projects/be6239167c474888803de4aa4039c40e/users/6109db02b8a6424da5c67098cd8806bb/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:31.917081 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-207adba1-6300-48d8-af59-88ebff7083f6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=be6239167c474888803de4aa4039c40e, user_id=6109db02b8a6424da5c67098cd8806bb, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:31.919795 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-207adba1-6300-48d8-af59-88ebff7083f6 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:31.938811 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1108/2226] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:57:31 2022] PUT /identity/v3/projects/be6239167c474888803de4aa4039c40e/users/6109db02b8a6424da5c67098cd8806bb/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:57:31.940936 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e1b302-e322-4022-8816-902d9bc09851 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:31.956776 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e1b302-e322-4022-8816-902d9bc09851 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:31.970139 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:57:32.054468 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c5385ee-3383-4bbf-b1d3-18362603b972 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:32.055697 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5385ee-3383-4bbf-b1d3-18362603b972 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:32.056050 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5385ee-3383-4bbf-b1d3-18362603b972 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:32.056401 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5385ee-3383-4bbf-b1d3-18362603b972 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:32.057888 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c5385ee-3383-4bbf-b1d3-18362603b972 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:32.061182 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c5385ee-3383-4bbf-b1d3-18362603b972 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:32.078311 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1119/2227] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:57:31 2022] GET /identity/v3/roles => generated 1420 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:57:32.085128 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-207adba1-6300-48d8-af59-88ebff7083f6 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:32.098712 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-207adba1-6300-48d8-af59-88ebff7083f6 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:32.105310 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e7f6a4-22b3-443c-813e-de609b8376b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:32.106322 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e7f6a4-22b3-443c-813e-de609b8376b9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:32.106924 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e7f6a4-22b3-443c-813e-de609b8376b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:32.107291 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e7f6a4-22b3-443c-813e-de609b8376b9 admin admin] PATH_INFO: `/v3/projects/be6239167c474888803de4aa4039c40e/users/6109db02b8a6424da5c67098cd8806bb/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:32.110582 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46e7f6a4-22b3-443c-813e-de609b8376b9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=be6239167c474888803de4aa4039c40e, user_id=6109db02b8a6424da5c67098cd8806bb, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:32.114980 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46e7f6a4-22b3-443c-813e-de609b8376b9 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:32.140140 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1109/2228] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:57:32 2022] PUT /identity/v3/projects/be6239167c474888803de4aa4039c40e/users/6109db02b8a6424da5c67098cd8806bb/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 00:57:32.142498 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59cfea20-a48f-4e8c-8bc7-65a4604474be None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:32.143392 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59cfea20-a48f-4e8c-8bc7-65a4604474be None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:32.143874 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59cfea20-a48f-4e8c-8bc7-65a4604474be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:32.195074 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-59cfea20-a48f-4e8c-8bc7-65a4604474be None None] MFA Rules not processed for user `6109db02b8a6424da5c67098cd8806bb`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:57:32.196442 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-59cfea20-a48f-4e8c-8bc7-65a4604474be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:32.322357 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1120/2229] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:57:32 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:57:32.337561 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e7f6a4-22b3-443c-813e-de609b8376b9 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:32.343065 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-46e7f6a4-22b3-443c-813e-de609b8376b9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:32.548008 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:57:32.569400 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e7f6a4-22b3-443c-813e-de609b8376b9 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:32.573356 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59cfea20-a48f-4e8c-8bc7-65a4604474be None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:32.576444 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-59cfea20-a48f-4e8c-8bc7-65a4604474be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:32.607505 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e43d3d-fea1-4c09-9625-dc04dfe8a9f0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:32.609459 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e43d3d-fea1-4c09-9625-dc04dfe8a9f0 service cinder] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:32.609739 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e43d3d-fea1-4c09-9625-dc04dfe8a9f0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:32.612356 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e43d3d-fea1-4c09-9625-dc04dfe8a9f0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:32.642383 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e43d3d-fea1-4c09-9625-dc04dfe8a9f0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:32.655459 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e43d3d-fea1-4c09-9625-dc04dfe8a9f0 service cinder] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:32.706132 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1110/2230] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:57:32 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:57:32.780963 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59cfea20-a48f-4e8c-8bc7-65a4604474be None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:32.812786 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d4c2464-25ea-400d-8a7c-844d863daf14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:32.813463 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d4c2464-25ea-400d-8a7c-844d863daf14 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:32.813745 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d4c2464-25ea-400d-8a7c-844d863daf14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:32.814096 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d4c2464-25ea-400d-8a7c-844d863daf14 admin admin] PATH_INFO: `/v3/users/a9af3af9452b474d887ab282ef684640` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:32.818715 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d4c2464-25ea-400d-8a7c-844d863daf14 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a9af3af9452b474d887ab282ef684640)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:32.827932 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d4c2464-25ea-400d-8a7c-844d863daf14 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:32.865249 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e43d3d-fea1-4c09-9625-dc04dfe8a9f0 service cinder] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:32.868425 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-e7e43d3d-fea1-4c09-9625-dc04dfe8a9f0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:32.882953 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e43d3d-fea1-4c09-9625-dc04dfe8a9f0 service cinder] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:32.915627 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d4cfac-f5aa-42a2-a72e-4ef4f0dec4cc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:32.917809 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d4cfac-f5aa-42a2-a72e-4ef4f0dec4cc service cinder] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:32.918547 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d4cfac-f5aa-42a2-a72e-4ef4f0dec4cc service cinder] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:32.919986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d4cfac-f5aa-42a2-a72e-4ef4f0dec4cc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:32.945630 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:57:32.980920 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-8d4c2464-25ea-400d-8a7c-844d863daf14 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a9af3af9452b474d887ab282ef684640'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:32.981728 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-8d4c2464-25ea-400d-8a7c-844d863daf14 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a9af3af9452b474d887ab282ef684640'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:32.997093 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-8d4c2464-25ea-400d-8a7c-844d863daf14 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a9af3af9452b474d887ab282ef684640'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:32.997982 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-8d4c2464-25ea-400d-8a7c-844d863daf14 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a9af3af9452b474d887ab282ef684640'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:33.017795 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-8d4c2464-25ea-400d-8a7c-844d863daf14 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a9af3af9452b474d887ab282ef684640'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:33.048565 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5d4cfac-f5aa-42a2-a72e-4ef4f0dec4cc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:33.050382 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5d4cfac-f5aa-42a2-a72e-4ef4f0dec4cc service cinder] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:33.055401 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1121/2231] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:57:32 2022] DELETE /identity/v3/users/a9af3af9452b474d887ab282ef684640 => generated 0 bytes in 482 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:57:33.063557 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45050dd1-38be-4d96-b029-6e184633f67b None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:33.064209 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45050dd1-38be-4d96-b029-6e184633f67b None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:33.064584 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45050dd1-38be-4d96-b029-6e184633f67b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:33.071239 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1111/2232] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:57:32 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 209 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 00:57:33.100075 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-45050dd1-38be-4d96-b029-6e184633f67b None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:57:33.101640 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-45050dd1-38be-4d96-b029-6e184633f67b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:33.257348 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1122/2233] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:57:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:57:33.268427 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d4cfac-f5aa-42a2-a72e-4ef4f0dec4cc service cinder] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:33.270258 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-e5d4cfac-f5aa-42a2-a72e-4ef4f0dec4cc service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:33.429106 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d4cfac-f5aa-42a2-a72e-4ef4f0dec4cc service cinder] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:33.527899 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172f0761-978a-4d0d-b455-99acafa31e45 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:33.528518 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172f0761-978a-4d0d-b455-99acafa31e45 service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:33.528777 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172f0761-978a-4d0d-b455-99acafa31e45 service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:33.529042 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172f0761-978a-4d0d-b455-99acafa31e45 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:33.547926 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-172f0761-978a-4d0d-b455-99acafa31e45 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:33.550007 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-172f0761-978a-4d0d-b455-99acafa31e45 service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:33.562845 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1112/2234] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:57:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 295 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:57:33.587016 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4724fd48-6807-4cc3-b65c-ad4647a133e5 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:33.587298 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4724fd48-6807-4cc3-b65c-ad4647a133e5 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:33.587632 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4724fd48-6807-4cc3-b65c-ad4647a133e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:33.642745 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-4724fd48-6807-4cc3-b65c-ad4647a133e5 None None] MFA Rules not processed for user `c19b10ebc31244dbb2b1539d94ac281d`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:57:33.644596 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-4724fd48-6807-4cc3-b65c-ad4647a133e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:33.764602 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1123/2235] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:57:33 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:57:33.766037 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172f0761-978a-4d0d-b455-99acafa31e45 service neutron] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:33.767616 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-172f0761-978a-4d0d-b455-99acafa31e45 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:33.811378 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172f0761-978a-4d0d-b455-99acafa31e45 service neutron] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:33.817624 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b921edaa-edb1-4648-b47b-de04b958c9cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:33.818299 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b921edaa-edb1-4648-b47b-de04b958c9cd admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:33.818637 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b921edaa-edb1-4648-b47b-de04b958c9cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:33.820501 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b921edaa-edb1-4648-b47b-de04b958c9cd admin admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:33.821178 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b921edaa-edb1-4648-b47b-de04b958c9cd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:33.824857 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b921edaa-edb1-4648-b47b-de04b958c9cd admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:33.889989 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1113/2236] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:57:33 2022] POST /identity/v3/projects => generated 377 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:57:33.894284 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4724fd48-6807-4cc3-b65c-ad4647a133e5 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:33.912209 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4724fd48-6807-4cc3-b65c-ad4647a133e5 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:33.933709 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b921edaa-edb1-4648-b47b-de04b958c9cd admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:33.935591 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-b921edaa-edb1-4648-b47b-de04b958c9cd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:33.980407 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:57:33.992952 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:57:34.042135 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f10ecf01-5411-4247-8693-0327c161035d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:34.043143 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10ecf01-5411-4247-8693-0327c161035d admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:34.043143 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10ecf01-5411-4247-8693-0327c161035d admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:34.043323 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10ecf01-5411-4247-8693-0327c161035d admin admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:34.043908 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f10ecf01-5411-4247-8693-0327c161035d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:34.045901 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f10ecf01-5411-4247-8693-0327c161035d admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:34.065537 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b921edaa-edb1-4648-b47b-de04b958c9cd admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:34.073531 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971bf70f-4172-41b1-9308-3510b6bcfe58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:34.075063 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971bf70f-4172-41b1-9308-3510b6bcfe58 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:34.075654 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971bf70f-4172-41b1-9308-3510b6bcfe58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:34.076139 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971bf70f-4172-41b1-9308-3510b6bcfe58 admin admin] PATH_INFO: `/v3/projects/f8f3e47fa0e04e77a29f2a3ef7566c80` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:34.078119 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-971bf70f-4172-41b1-9308-3510b6bcfe58 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f8f3e47fa0e04e77a29f2a3ef7566c80)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:34.081215 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-971bf70f-4172-41b1-9308-3510b6bcfe58 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:34.085294 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1124/2237] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:57:33 2022] POST /identity/v3/users => generated 295 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:57:34.088308 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f10ecf01-5411-4247-8693-0327c161035d admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:34.100050 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f10ecf01-5411-4247-8693-0327c161035d admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:34.108927 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f38622-df92-42cb-82bd-6cc13e75b068 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:34.110795 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f38622-df92-42cb-82bd-6cc13e75b068 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:34.111355 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f38622-df92-42cb-82bd-6cc13e75b068 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:34.111860 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f38622-df92-42cb-82bd-6cc13e75b068 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:34.114231 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f38622-df92-42cb-82bd-6cc13e75b068 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:34.117926 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f38622-df92-42cb-82bd-6cc13e75b068 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:34.132056 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1125/2238] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:57:34 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 00:57:34.136103 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f38622-df92-42cb-82bd-6cc13e75b068 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:34.147908 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f38622-df92-42cb-82bd-6cc13e75b068 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:34.149729 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-971bf70f-4172-41b1-9308-3510b6bcfe58 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f8f3e47fa0e04e77a29f2a3ef7566c80', 'request_id': 'req-971bf70f-4172-41b1-9308-3510b6bcfe58'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:34.161982 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-971bf70f-4172-41b1-9308-3510b6bcfe58 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f8f3e47fa0e04e77a29f2a3ef7566c80', 'request_id': 'req-971bf70f-4172-41b1-9308-3510b6bcfe58'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:34.171074 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-971bf70f-4172-41b1-9308-3510b6bcfe58 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f8f3e47fa0e04e77a29f2a3ef7566c80', 'request_id': 'req-971bf70f-4172-41b1-9308-3510b6bcfe58'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:34.198776 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-971bf70f-4172-41b1-9308-3510b6bcfe58 admin admin] The token cache is being invalidate because project f8f3e47fa0e04e77a29f2a3ef7566c80 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81386) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:57:34.199298 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-971bf70f-4172-41b1-9308-3510b6bcfe58 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:34.200236 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-971bf70f-4172-41b1-9308-3510b6bcfe58 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:34.202475 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1114/2239] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:57:33 2022] DELETE /identity/v3/projects/f8f3e47fa0e04e77a29f2a3ef7566c80 => generated 0 bytes in 269 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:57:34.257819 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9bc3f7c-2dc4-493e-9898-7c4ac51f40f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:34.258714 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9bc3f7c-2dc4-493e-9898-7c4ac51f40f9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:34.259344 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9bc3f7c-2dc4-493e-9898-7c4ac51f40f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:34.259725 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9bc3f7c-2dc4-493e-9898-7c4ac51f40f9 admin admin] PATH_INFO: `/v3/projects/adf4f7abd15d45d1a06405bdb1e94eef/users/919d4aec9bee4ef99aef27e7546bbe55/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:34.283114 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9bc3f7c-2dc4-493e-9898-7c4ac51f40f9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=adf4f7abd15d45d1a06405bdb1e94eef, user_id=919d4aec9bee4ef99aef27e7546bbe55, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:34.286523 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9bc3f7c-2dc4-493e-9898-7c4ac51f40f9 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:34.338456 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971bf70f-4172-41b1-9308-3510b6bcfe58 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:34.339782 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-971bf70f-4172-41b1-9308-3510b6bcfe58 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:34.366346 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971bf70f-4172-41b1-9308-3510b6bcfe58 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:34.370225 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1126/2240] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:57:34 2022] PUT /identity/v3/projects/adf4f7abd15d45d1a06405bdb1e94eef/users/919d4aec9bee4ef99aef27e7546bbe55/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 00:57:34.498892 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bbdf149-6f17-4c13-ad08-7d48b4c04825 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:34.499765 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bbdf149-6f17-4c13-ad08-7d48b4c04825 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:34.500100 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bbdf149-6f17-4c13-ad08-7d48b4c04825 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:34.500430 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bbdf149-6f17-4c13-ad08-7d48b4c04825 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:34.501137 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bbdf149-6f17-4c13-ad08-7d48b4c04825 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:34.502901 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bbdf149-6f17-4c13-ad08-7d48b4c04825 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:34.514712 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1115/2241] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:57:34 2022] GET /identity/v3/roles => generated 1420 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 00:57:34.518844 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9bc3f7c-2dc4-493e-9898-7c4ac51f40f9 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:34.534109 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9bc3f7c-2dc4-493e-9898-7c4ac51f40f9 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:34.540243 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f06f1db-1275-4ede-82ce-37a2d1cf66b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:34.541345 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f06f1db-1275-4ede-82ce-37a2d1cf66b7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:34.541817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f06f1db-1275-4ede-82ce-37a2d1cf66b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:34.542238 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f06f1db-1275-4ede-82ce-37a2d1cf66b7 admin admin] PATH_INFO: `/v3/projects/adf4f7abd15d45d1a06405bdb1e94eef/users/919d4aec9bee4ef99aef27e7546bbe55/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:34.548449 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f06f1db-1275-4ede-82ce-37a2d1cf66b7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=adf4f7abd15d45d1a06405bdb1e94eef, user_id=919d4aec9bee4ef99aef27e7546bbe55, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:34.555624 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f06f1db-1275-4ede-82ce-37a2d1cf66b7 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:34.571973 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1127/2242] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:57:34 2022] PUT /identity/v3/projects/adf4f7abd15d45d1a06405bdb1e94eef/users/919d4aec9bee4ef99aef27e7546bbe55/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:57:39.520941 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7455ef35-d135-4519-8b58-025c24b185e9 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:39.521468 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7455ef35-d135-4519-8b58-025c24b185e9 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:39.521560 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7455ef35-d135-4519-8b58-025c24b185e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:39.569747 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-7455ef35-d135-4519-8b58-025c24b185e9 None None] MFA Rules not processed for user `919d4aec9bee4ef99aef27e7546bbe55`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:57:39.570861 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-7455ef35-d135-4519-8b58-025c24b185e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:39.636521 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:57:39.706249 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1116/2243] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:57:39 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:57:39.715680 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f06f1db-1275-4ede-82ce-37a2d1cf66b7 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:39.718764 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-1f06f1db-1275-4ede-82ce-37a2d1cf66b7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:39.885597 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f06f1db-1275-4ede-82ce-37a2d1cf66b7 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:39.892314 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ffc4bfc-c8bc-4464-a5f8-3200ac2c48da service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f142988a4dd14a5f96eacea45c046f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:39.892932 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ffc4bfc-c8bc-4464-a5f8-3200ac2c48da service nova] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:39.893211 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ffc4bfc-c8bc-4464-a5f8-3200ac2c48da service nova] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:39.893508 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ffc4bfc-c8bc-4464-a5f8-3200ac2c48da service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:39.895487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-5ffc4bfc-c8bc-4464-a5f8-3200ac2c48da service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:39.938711 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:57:40.011424 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ffc4bfc-c8bc-4464-a5f8-3200ac2c48da service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:40.011424 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ffc4bfc-c8bc-4464-a5f8-3200ac2c48da service nova] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:40.031771 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1128/2244] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:57:39 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:57:49.693074 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=11 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:57:50.022266 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=15 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:57:59.570223 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7455ef35-d135-4519-8b58-025c24b185e9 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:59.571922 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-7455ef35-d135-4519-8b58-025c24b185e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:59.644139 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7455ef35-d135-4519-8b58-025c24b185e9 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:59.649027 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3efb5273-4afe-4ab0-848a-25fa1a74d427 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:59.649699 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efb5273-4afe-4ab0-848a-25fa1a74d427 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:59.649979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efb5273-4afe-4ab0-848a-25fa1a74d427 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:59.650248 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efb5273-4afe-4ab0-848a-25fa1a74d427 admin admin] PATH_INFO: `/v3/users/b1b7c24458214f77a9d447bbd22e70d1` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:59.651521 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3efb5273-4afe-4ab0-848a-25fa1a74d427 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b1b7c24458214f77a9d447bbd22e70d1)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:59.653505 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3efb5273-4afe-4ab0-848a-25fa1a74d427 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:59.702862 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-3efb5273-4afe-4ab0-848a-25fa1a74d427 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b1b7c24458214f77a9d447bbd22e70d1'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:59.711498 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-3efb5273-4afe-4ab0-848a-25fa1a74d427 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b1b7c24458214f77a9d447bbd22e70d1'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:59.712254 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-3efb5273-4afe-4ab0-848a-25fa1a74d427 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b1b7c24458214f77a9d447bbd22e70d1'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:59.719225 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-3efb5273-4afe-4ab0-848a-25fa1a74d427 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b1b7c24458214f77a9d447bbd22e70d1'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:59.719785 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-3efb5273-4afe-4ab0-848a-25fa1a74d427 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b1b7c24458214f77a9d447bbd22e70d1'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:57:59.737025 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1117/2245] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:57:59 2022] DELETE /identity/v3/users/b1b7c24458214f77a9d447bbd22e70d1 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:57:59.742617 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cedd2be-ae4d-48c5-a949-87e87f71a37b None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:59.743143 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cedd2be-ae4d-48c5-a949-87e87f71a37b None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:59.743143 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cedd2be-ae4d-48c5-a949-87e87f71a37b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:59.762583 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-9cedd2be-ae4d-48c5-a949-87e87f71a37b None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:57:59.763777 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-9cedd2be-ae4d-48c5-a949-87e87f71a37b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:59.825786 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1129/2246] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:57:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:57:59.835979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3efb5273-4afe-4ab0-848a-25fa1a74d427 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:57:59.837489 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-3efb5273-4afe-4ab0-848a-25fa1a74d427 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:57:59.912122 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3efb5273-4afe-4ab0-848a-25fa1a74d427 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:57:59.917989 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df9837f9-e54c-42ba-8254-170eab5cb0a3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:57:59.917989 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9837f9-e54c-42ba-8254-170eab5cb0a3 service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:57:59.918280 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9837f9-e54c-42ba-8254-170eab5cb0a3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:57:59.918484 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9837f9-e54c-42ba-8254-170eab5cb0a3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:57:59.929437 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df9837f9-e54c-42ba-8254-170eab5cb0a3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:57:59.930887 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df9837f9-e54c-42ba-8254-170eab5cb0a3 service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:57:59.942511 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1118/2247] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:57:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:58:00.240104 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cedd2be-ae4d-48c5-a949-87e87f71a37b None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:00.241745 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-9cedd2be-ae4d-48c5-a949-87e87f71a37b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:00.254920 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cedd2be-ae4d-48c5-a949-87e87f71a37b None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:00.262262 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c84ae7b7-8c6a-44fc-9268-12da99abb0f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:00.263051 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c84ae7b7-8c6a-44fc-9268-12da99abb0f6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:00.263446 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c84ae7b7-8c6a-44fc-9268-12da99abb0f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:00.263807 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c84ae7b7-8c6a-44fc-9268-12da99abb0f6 admin admin] PATH_INFO: `/v3/projects/7750b9ec659c40a1aafc98b0107032c5` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:00.265121 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c84ae7b7-8c6a-44fc-9268-12da99abb0f6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7750b9ec659c40a1aafc98b0107032c5)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:00.267089 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c84ae7b7-8c6a-44fc-9268-12da99abb0f6 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:00.314236 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-c84ae7b7-8c6a-44fc-9268-12da99abb0f6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7750b9ec659c40a1aafc98b0107032c5', 'request_id': 'req-c84ae7b7-8c6a-44fc-9268-12da99abb0f6'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:00.324292 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-c84ae7b7-8c6a-44fc-9268-12da99abb0f6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7750b9ec659c40a1aafc98b0107032c5', 'request_id': 'req-c84ae7b7-8c6a-44fc-9268-12da99abb0f6'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:00.330614 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-c84ae7b7-8c6a-44fc-9268-12da99abb0f6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7750b9ec659c40a1aafc98b0107032c5', 'request_id': 'req-c84ae7b7-8c6a-44fc-9268-12da99abb0f6'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:00.342429 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-c84ae7b7-8c6a-44fc-9268-12da99abb0f6 admin admin] The token cache is being invalidate because project 7750b9ec659c40a1aafc98b0107032c5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:58:00.342713 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-c84ae7b7-8c6a-44fc-9268-12da99abb0f6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:00.343356 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-c84ae7b7-8c6a-44fc-9268-12da99abb0f6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:00.344831 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1130/2248] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:58:00 2022] DELETE /identity/v3/projects/7750b9ec659c40a1aafc98b0107032c5 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:58:00.350984 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-532b5c75-a7c7-43d2-8798-f9a89a509145 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:00.351275 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-532b5c75-a7c7-43d2-8798-f9a89a509145 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:00.351538 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-532b5c75-a7c7-43d2-8798-f9a89a509145 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:00.372028 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-532b5c75-a7c7-43d2-8798-f9a89a509145 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:58:00.373244 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-532b5c75-a7c7-43d2-8798-f9a89a509145 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:00.394468 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:58:00.450658 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1119/2249] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:58:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:58:00.459319 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d2838a-d900-44b3-bd91-31bfe223ee48 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:00.459655 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d2838a-d900-44b3-bd91-31bfe223ee48 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:00.459936 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d2838a-d900-44b3-bd91-31bfe223ee48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:00.480642 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-23d2838a-d900-44b3-bd91-31bfe223ee48 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:58:00.482656 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-23d2838a-d900-44b3-bd91-31bfe223ee48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:00.492190 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1131/2250] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:58:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:58:00.501303 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-532b5c75-a7c7-43d2-8798-f9a89a509145 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:00.513843 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-532b5c75-a7c7-43d2-8798-f9a89a509145 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:00.518498 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe09d91-afd4-4f09-8844-bf519cbf9de1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:00.519701 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe09d91-afd4-4f09-8844-bf519cbf9de1 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:00.519950 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe09d91-afd4-4f09-8844-bf519cbf9de1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:00.520212 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe09d91-afd4-4f09-8844-bf519cbf9de1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:00.520859 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abe09d91-afd4-4f09-8844-bf519cbf9de1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:58:00.521156 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abe09d91-afd4-4f09-8844-bf519cbf9de1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:00.522843 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abe09d91-afd4-4f09-8844-bf519cbf9de1 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:00.534774 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1120/2251] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 00:58:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:58:00.539161 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23d2838a-d900-44b3-bd91-31bfe223ee48 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:00.548685 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23d2838a-d900-44b3-bd91-31bfe223ee48 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:00.553826 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aaf52b9-75b5-4110-9322-c3a850b71281 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:00.554431 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aaf52b9-75b5-4110-9322-c3a850b71281 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:00.554707 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aaf52b9-75b5-4110-9322-c3a850b71281 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:00.554973 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aaf52b9-75b5-4110-9322-c3a850b71281 admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:00.555615 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aaf52b9-75b5-4110-9322-c3a850b71281 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:00.557530 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aaf52b9-75b5-4110-9322-c3a850b71281 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:00.597587 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1132/2252] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:58:00 2022] POST /identity/v3/projects => generated 383 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:58:00.599427 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe09d91-afd4-4f09-8844-bf519cbf9de1 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:00.613861 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe09d91-afd4-4f09-8844-bf519cbf9de1 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:00.639799 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aaf52b9-75b5-4110-9322-c3a850b71281 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:00.642430 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-3aaf52b9-75b5-4110-9322-c3a850b71281 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:00.694358 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dbcb819-c49f-4c9f-98b3-a2d98820971c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:00.695123 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dbcb819-c49f-4c9f-98b3-a2d98820971c admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:00.695646 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dbcb819-c49f-4c9f-98b3-a2d98820971c admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:00.696050 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dbcb819-c49f-4c9f-98b3-a2d98820971c admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:00.696904 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dbcb819-c49f-4c9f-98b3-a2d98820971c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:00.699008 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dbcb819-c49f-4c9f-98b3-a2d98820971c admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:00.719509 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:58:00.723948 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1121/2253] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:58:00 2022] POST /identity/v3/users => generated 292 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:58:00.728497 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dbcb819-c49f-4c9f-98b3-a2d98820971c admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:00.731607 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aaf52b9-75b5-4110-9322-c3a850b71281 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:00.739882 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f161ecc-5d49-4c89-b50b-fab8c7ab2257 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:00.740626 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f161ecc-5d49-4c89-b50b-fab8c7ab2257 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:00.740968 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f161ecc-5d49-4c89-b50b-fab8c7ab2257 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:00.741341 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f161ecc-5d49-4c89-b50b-fab8c7ab2257 admin admin] PATH_INFO: `/v3/users/1dd2549f4f29404f92abbdc086d48ac7` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:00.742977 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f161ecc-5d49-4c89-b50b-fab8c7ab2257 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1dd2549f4f29404f92abbdc086d48ac7)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:00.744789 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dbcb819-c49f-4c9f-98b3-a2d98820971c admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:00.746357 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f161ecc-5d49-4c89-b50b-fab8c7ab2257 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:00.751363 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b36529b2-e35a-4e6b-b4b8-94a0ce400c8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:00.752102 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36529b2-e35a-4e6b-b4b8-94a0ce400c8d admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:00.752383 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36529b2-e35a-4e6b-b4b8-94a0ce400c8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:00.752648 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36529b2-e35a-4e6b-b4b8-94a0ce400c8d admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:00.753957 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b36529b2-e35a-4e6b-b4b8-94a0ce400c8d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:00.756385 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b36529b2-e35a-4e6b-b4b8-94a0ce400c8d admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:00.773008 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1122/2254] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:58:00 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:58:00.776409 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b36529b2-e35a-4e6b-b4b8-94a0ce400c8d admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:00.797348 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b36529b2-e35a-4e6b-b4b8-94a0ce400c8d admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:00.804453 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f5472e5-5f77-4002-be6c-238c92c44980 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:00.805578 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f5472e5-5f77-4002-be6c-238c92c44980 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:00.806249 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f5472e5-5f77-4002-be6c-238c92c44980 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:00.806711 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f5472e5-5f77-4002-be6c-238c92c44980 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:00.807585 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f5472e5-5f77-4002-be6c-238c92c44980 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:00.812577 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f5472e5-5f77-4002-be6c-238c92c44980 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:00.826161 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1123/2255] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:58:00 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 00:58:00.830017 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f5472e5-5f77-4002-be6c-238c92c44980 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:00.838290 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-6f161ecc-5d49-4c89-b50b-fab8c7ab2257 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1dd2549f4f29404f92abbdc086d48ac7'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:00.839523 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-6f161ecc-5d49-4c89-b50b-fab8c7ab2257 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1dd2549f4f29404f92abbdc086d48ac7'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:00.843848 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f5472e5-5f77-4002-be6c-238c92c44980 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:00.850447 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e3a8a0-1b7b-4a71-bc17-0294d7ac8306 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:00.851172 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e3a8a0-1b7b-4a71-bc17-0294d7ac8306 admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:00.851447 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e3a8a0-1b7b-4a71-bc17-0294d7ac8306 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:00.851711 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e3a8a0-1b7b-4a71-bc17-0294d7ac8306 admin admin] PATH_INFO: `/v3/projects/b161c2e2dd1f42159dc0d71e8216516c/users/80dd52f1455f44b587716a41e632a155/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:00.853312 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-6f161ecc-5d49-4c89-b50b-fab8c7ab2257 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1dd2549f4f29404f92abbdc086d48ac7'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:00.854535 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-6f161ecc-5d49-4c89-b50b-fab8c7ab2257 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1dd2549f4f29404f92abbdc086d48ac7'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:00.867233 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-6f161ecc-5d49-4c89-b50b-fab8c7ab2257 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1dd2549f4f29404f92abbdc086d48ac7'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:00.870446 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0e3a8a0-1b7b-4a71-bc17-0294d7ac8306 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b161c2e2dd1f42159dc0d71e8216516c, user_id=80dd52f1455f44b587716a41e632a155, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:00.873183 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0e3a8a0-1b7b-4a71-bc17-0294d7ac8306 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:00.888646 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1133/2256] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:58:00 2022] DELETE /identity/v3/users/1dd2549f4f29404f92abbdc086d48ac7 => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 00:58:00.892999 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21770f2b-67ea-46bb-b02b-b5049ffb9c95 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:00.893193 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21770f2b-67ea-46bb-b02b-b5049ffb9c95 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:00.893461 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21770f2b-67ea-46bb-b02b-b5049ffb9c95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:00.904269 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1124/2257] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:58:00 2022] PUT /identity/v3/projects/b161c2e2dd1f42159dc0d71e8216516c/users/80dd52f1455f44b587716a41e632a155/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 75 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:58:00.914998 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-21770f2b-67ea-46bb-b02b-b5049ffb9c95 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:58:00.915574 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dadf0b8-55d2-431f-b950-54ba2ad53bcb None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:00.915930 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dadf0b8-55d2-431f-b950-54ba2ad53bcb None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:00.916280 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dadf0b8-55d2-431f-b950-54ba2ad53bcb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:00.917069 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-21770f2b-67ea-46bb-b02b-b5049ffb9c95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:00.937210 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-6dadf0b8-55d2-431f-b950-54ba2ad53bcb None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:58:00.938451 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-6dadf0b8-55d2-431f-b950-54ba2ad53bcb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:00.997237 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1134/2258] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:58:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 00:58:01.008455 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21770f2b-67ea-46bb-b02b-b5049ffb9c95 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:01.010619 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-21770f2b-67ea-46bb-b02b-b5049ffb9c95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:01.016622 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:58:01.036118 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1125/2259] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:58:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:58:01.060332 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dadf0b8-55d2-431f-b950-54ba2ad53bcb None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:01.061896 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-6dadf0b8-55d2-431f-b950-54ba2ad53bcb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:01.077524 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21770f2b-67ea-46bb-b02b-b5049ffb9c95 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:01.081140 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b0469c1-e056-46d5-985b-06ccc181d65a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:01.082034 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0469c1-e056-46d5-985b-06ccc181d65a service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:01.082380 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0469c1-e056-46d5-985b-06ccc181d65a service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:01.082657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0469c1-e056-46d5-985b-06ccc181d65a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:01.091885 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b0469c1-e056-46d5-985b-06ccc181d65a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:01.093361 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b0469c1-e056-46d5-985b-06ccc181d65a service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:01.103570 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1135/2260] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:58:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 00:58:01.125935 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dadf0b8-55d2-431f-b950-54ba2ad53bcb None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:01.130092 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cfc2cc8-dcd8-4b20-8421-e2ccda589df5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:01.130703 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cfc2cc8-dcd8-4b20-8421-e2ccda589df5 service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:01.130981 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cfc2cc8-dcd8-4b20-8421-e2ccda589df5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:01.131250 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cfc2cc8-dcd8-4b20-8421-e2ccda589df5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:01.141814 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cfc2cc8-dcd8-4b20-8421-e2ccda589df5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:01.143379 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cfc2cc8-dcd8-4b20-8421-e2ccda589df5 service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:01.153917 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1126/2261] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:58:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 00:58:01.564168 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b0469c1-e056-46d5-985b-06ccc181d65a service neutron] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:01.567231 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-9b0469c1-e056-46d5-985b-06ccc181d65a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:01.586106 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b0469c1-e056-46d5-985b-06ccc181d65a service neutron] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:01.592273 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36fc4b95-819f-42e0-b6e2-1598ccc709d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:01.592945 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36fc4b95-819f-42e0-b6e2-1598ccc709d1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:01.593417 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36fc4b95-819f-42e0-b6e2-1598ccc709d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:01.593785 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36fc4b95-819f-42e0-b6e2-1598ccc709d1 admin admin] PATH_INFO: `/v3/projects/4e348390b8774830a7542bb983bd5672` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:01.598628 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36fc4b95-819f-42e0-b6e2-1598ccc709d1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4e348390b8774830a7542bb983bd5672)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:01.598901 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36fc4b95-819f-42e0-b6e2-1598ccc709d1 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:01.619582 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:58:01.675172 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-36fc4b95-819f-42e0-b6e2-1598ccc709d1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4e348390b8774830a7542bb983bd5672', 'request_id': 'req-36fc4b95-819f-42e0-b6e2-1598ccc709d1'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:01.695417 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-36fc4b95-819f-42e0-b6e2-1598ccc709d1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4e348390b8774830a7542bb983bd5672', 'request_id': 'req-36fc4b95-819f-42e0-b6e2-1598ccc709d1'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:01.702574 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-36fc4b95-819f-42e0-b6e2-1598ccc709d1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4e348390b8774830a7542bb983bd5672', 'request_id': 'req-36fc4b95-819f-42e0-b6e2-1598ccc709d1'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:01.718043 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-36fc4b95-819f-42e0-b6e2-1598ccc709d1 admin admin] The token cache is being invalidate because project 4e348390b8774830a7542bb983bd5672 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:58:01.718456 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-36fc4b95-819f-42e0-b6e2-1598ccc709d1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:01.719558 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-36fc4b95-819f-42e0-b6e2-1598ccc709d1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:01.722545 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1136/2262] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:58:01 2022] DELETE /identity/v3/projects/4e348390b8774830a7542bb983bd5672 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 00:58:01.727613 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de94fe84-e919-47b7-8699-ef80cdb6edeb None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:01.728123 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de94fe84-e919-47b7-8699-ef80cdb6edeb None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:01.728123 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de94fe84-e919-47b7-8699-ef80cdb6edeb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:01.756345 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-de94fe84-e919-47b7-8699-ef80cdb6edeb None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:58:01.758079 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-de94fe84-e919-47b7-8699-ef80cdb6edeb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:01.837172 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1127/2263] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:58:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:58:01.845555 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e97e5e3-b550-42c3-ad4d-1fb7f1843d9c None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:01.846141 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e97e5e3-b550-42c3-ad4d-1fb7f1843d9c None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:01.846141 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e97e5e3-b550-42c3-ad4d-1fb7f1843d9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:01.868847 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-2e97e5e3-b550-42c3-ad4d-1fb7f1843d9c None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:58:01.870028 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-2e97e5e3-b550-42c3-ad4d-1fb7f1843d9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:01.888689 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1137/2264] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:58:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:58:01.890266 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de94fe84-e919-47b7-8699-ef80cdb6edeb None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:01.902030 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de94fe84-e919-47b7-8699-ef80cdb6edeb None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:01.909050 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e475b9cf-2a68-40da-b658-ce006df58089 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:01.909822 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e475b9cf-2a68-40da-b658-ce006df58089 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:01.910165 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e475b9cf-2a68-40da-b658-ce006df58089 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:01.910501 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e475b9cf-2a68-40da-b658-ce006df58089 admin admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:01.911220 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e475b9cf-2a68-40da-b658-ce006df58089 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:58:01.911588 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e475b9cf-2a68-40da-b658-ce006df58089 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:01.913351 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e475b9cf-2a68-40da-b658-ce006df58089 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:01.927141 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1128/2265] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 00:58:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:58:01.931290 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e97e5e3-b550-42c3-ad4d-1fb7f1843d9c None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:01.947809 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e97e5e3-b550-42c3-ad4d-1fb7f1843d9c None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:01.953319 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689baacd-2285-40f6-a17e-d07beda88a52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:01.953961 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689baacd-2285-40f6-a17e-d07beda88a52 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:01.954224 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689baacd-2285-40f6-a17e-d07beda88a52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:01.955204 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689baacd-2285-40f6-a17e-d07beda88a52 admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:01.955204 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-689baacd-2285-40f6-a17e-d07beda88a52 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:01.957802 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-689baacd-2285-40f6-a17e-d07beda88a52 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:02.013510 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1138/2266] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:58:01 2022] POST /identity/v3/projects => generated 371 bytes in 83 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:58:02.017449 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e475b9cf-2a68-40da-b658-ce006df58089 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:02.030534 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e475b9cf-2a68-40da-b658-ce006df58089 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:02.163493 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973409f8-8d97-4a12-9af0-f78bf7ce8cc9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:02.164307 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973409f8-8d97-4a12-9af0-f78bf7ce8cc9 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:02.165111 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973409f8-8d97-4a12-9af0-f78bf7ce8cc9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:02.165247 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973409f8-8d97-4a12-9af0-f78bf7ce8cc9 admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:02.165888 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-973409f8-8d97-4a12-9af0-f78bf7ce8cc9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:02.168336 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-973409f8-8d97-4a12-9af0-f78bf7ce8cc9 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:02.199005 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1129/2267] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:58:02 2022] POST /identity/v3/users => generated 286 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:58:02.202754 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689baacd-2285-40f6-a17e-d07beda88a52 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:02.217584 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689baacd-2285-40f6-a17e-d07beda88a52 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:02.224126 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e059aeb-fde8-4c1b-8c32-31d75e772707 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:02.224937 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e059aeb-fde8-4c1b-8c32-31d75e772707 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:02.225367 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e059aeb-fde8-4c1b-8c32-31d75e772707 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:02.225739 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e059aeb-fde8-4c1b-8c32-31d75e772707 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:02.226510 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e059aeb-fde8-4c1b-8c32-31d75e772707 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:02.228314 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e059aeb-fde8-4c1b-8c32-31d75e772707 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:02.241943 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1139/2268] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:58:02 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 00:58:02.246429 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973409f8-8d97-4a12-9af0-f78bf7ce8cc9 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:02.259601 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973409f8-8d97-4a12-9af0-f78bf7ce8cc9 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:02.265782 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-215333f7-444a-4ba2-a96b-84ba9eee5088 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:02.266594 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215333f7-444a-4ba2-a96b-84ba9eee5088 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:02.266978 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215333f7-444a-4ba2-a96b-84ba9eee5088 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:02.267718 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215333f7-444a-4ba2-a96b-84ba9eee5088 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:02.268523 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-215333f7-444a-4ba2-a96b-84ba9eee5088 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:02.271216 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-215333f7-444a-4ba2-a96b-84ba9eee5088 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:02.290204 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1130/2269] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:58:02 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:58:02.294200 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e059aeb-fde8-4c1b-8c32-31d75e772707 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:02.307794 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e059aeb-fde8-4c1b-8c32-31d75e772707 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:02.314163 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a76224f-a376-469b-b1ab-06e43dc7fd1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:02.315423 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a76224f-a376-469b-b1ab-06e43dc7fd1e admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:02.316185 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a76224f-a376-469b-b1ab-06e43dc7fd1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:02.316790 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a76224f-a376-469b-b1ab-06e43dc7fd1e admin admin] PATH_INFO: `/v3/projects/d5f8f084099d4b37b67aba3cda9ecff3/users/89b7e5e8b24b4b96aa1b73efd77108a6/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:02.367843 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a76224f-a376-469b-b1ab-06e43dc7fd1e admin admin] RBAC: Authorizing `identity:create_grant(project_id=d5f8f084099d4b37b67aba3cda9ecff3, user_id=89b7e5e8b24b4b96aa1b73efd77108a6, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:02.373438 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a76224f-a376-469b-b1ab-06e43dc7fd1e admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:02.402169 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1140/2270] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:58:02 2022] PUT /identity/v3/projects/d5f8f084099d4b37b67aba3cda9ecff3/users/89b7e5e8b24b4b96aa1b73efd77108a6/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:58:02.497861 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8875015-d26a-47b5-81a4-640bd73bd0f8 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:02.498550 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8875015-d26a-47b5-81a4-640bd73bd0f8 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:02.498958 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8875015-d26a-47b5-81a4-640bd73bd0f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:02.515868 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:58:02.567547 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-a8875015-d26a-47b5-81a4-640bd73bd0f8 None None] MFA Rules not processed for user `89b7e5e8b24b4b96aa1b73efd77108a6`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:58:02.569147 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-a8875015-d26a-47b5-81a4-640bd73bd0f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:02.661642 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1131/2271] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:58:02 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:58:02.676477 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a76224f-a376-469b-b1ab-06e43dc7fd1e admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:02.680657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-6a76224f-a376-469b-b1ab-06e43dc7fd1e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:02.765249 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a76224f-a376-469b-b1ab-06e43dc7fd1e admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:02.769920 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ec40ef-2480-4670-8c9d-de1d30ed1a71 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:02.770540 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ec40ef-2480-4670-8c9d-de1d30ed1a71 service cinder] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:02.770819 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ec40ef-2480-4670-8c9d-de1d30ed1a71 service cinder] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:02.771085 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ec40ef-2480-4670-8c9d-de1d30ed1a71 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:02.780560 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9ec40ef-2480-4670-8c9d-de1d30ed1a71 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:02.782023 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9ec40ef-2480-4670-8c9d-de1d30ed1a71 service cinder] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:02.796396 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1141/2272] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:58:02 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:58:02.822820 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8875015-d26a-47b5-81a4-640bd73bd0f8 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:02.908114 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8875015-d26a-47b5-81a4-640bd73bd0f8 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:02.914340 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e1a82d4-e848-4235-8618-cc5b776ec6f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:02.915017 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1a82d4-e848-4235-8618-cc5b776ec6f8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:02.915360 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1a82d4-e848-4235-8618-cc5b776ec6f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:02.915697 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1a82d4-e848-4235-8618-cc5b776ec6f8 admin admin] PATH_INFO: `/v3/users/89b7e5e8b24b4b96aa1b73efd77108a6` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:02.916917 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e1a82d4-e848-4235-8618-cc5b776ec6f8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=89b7e5e8b24b4b96aa1b73efd77108a6)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:02.918957 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e1a82d4-e848-4235-8618-cc5b776ec6f8 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:02.986837 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-1e1a82d4-e848-4235-8618-cc5b776ec6f8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '89b7e5e8b24b4b96aa1b73efd77108a6'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:02.995957 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-1e1a82d4-e848-4235-8618-cc5b776ec6f8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '89b7e5e8b24b4b96aa1b73efd77108a6'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:02.997249 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-1e1a82d4-e848-4235-8618-cc5b776ec6f8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '89b7e5e8b24b4b96aa1b73efd77108a6'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:03.006908 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-1e1a82d4-e848-4235-8618-cc5b776ec6f8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '89b7e5e8b24b4b96aa1b73efd77108a6'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:03.007910 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-1e1a82d4-e848-4235-8618-cc5b776ec6f8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '89b7e5e8b24b4b96aa1b73efd77108a6'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:03.032795 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1132/2273] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:58:02 2022] DELETE /identity/v3/users/89b7e5e8b24b4b96aa1b73efd77108a6 => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:58:03.038424 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2e760b-c44e-4d5e-bee8-4d62056c78b5 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:03.038918 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2e760b-c44e-4d5e-bee8-4d62056c78b5 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:03.039148 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2e760b-c44e-4d5e-bee8-4d62056c78b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:03.060131 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-4b2e760b-c44e-4d5e-bee8-4d62056c78b5 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:58:03.062079 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-4b2e760b-c44e-4d5e-bee8-4d62056c78b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:03.105740 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:58:03.170199 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1142/2274] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:58:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:58:03.181299 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e1a82d4-e848-4235-8618-cc5b776ec6f8 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:03.182728 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-1e1a82d4-e848-4235-8618-cc5b776ec6f8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:03.205619 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:58:03.290917 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e1a82d4-e848-4235-8618-cc5b776ec6f8 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:03.294724 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1a7d0e-c3b1-4f64-90be-b5f78caafbfd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:03.295330 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1a7d0e-c3b1-4f64-90be-b5f78caafbfd service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:03.295606 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1a7d0e-c3b1-4f64-90be-b5f78caafbfd service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:03.295873 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1a7d0e-c3b1-4f64-90be-b5f78caafbfd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:03.305772 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c1a7d0e-c3b1-4f64-90be-b5f78caafbfd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:03.307216 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c1a7d0e-c3b1-4f64-90be-b5f78caafbfd service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:03.319908 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1133/2275] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:58:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:58:03.744887 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2e760b-c44e-4d5e-bee8-4d62056c78b5 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:03.746744 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-4b2e760b-c44e-4d5e-bee8-4d62056c78b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:03.791964 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2e760b-c44e-4d5e-bee8-4d62056c78b5 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:03.802615 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8449d620-d936-41ae-8779-263a1798eb0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:03.803537 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8449d620-d936-41ae-8779-263a1798eb0a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:03.803885 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8449d620-d936-41ae-8779-263a1798eb0a admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:03.804433 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8449d620-d936-41ae-8779-263a1798eb0a admin admin] PATH_INFO: `/v3/projects/d5f8f084099d4b37b67aba3cda9ecff3` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:03.805905 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8449d620-d936-41ae-8779-263a1798eb0a admin admin] RBAC: Authorizing `identity:delete_project(project_id=d5f8f084099d4b37b67aba3cda9ecff3)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:03.808757 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8449d620-d936-41ae-8779-263a1798eb0a admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:03.883391 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-8449d620-d936-41ae-8779-263a1798eb0a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd5f8f084099d4b37b67aba3cda9ecff3', 'request_id': 'req-8449d620-d936-41ae-8779-263a1798eb0a'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:03.892265 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-8449d620-d936-41ae-8779-263a1798eb0a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd5f8f084099d4b37b67aba3cda9ecff3', 'request_id': 'req-8449d620-d936-41ae-8779-263a1798eb0a'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:03.898652 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-8449d620-d936-41ae-8779-263a1798eb0a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd5f8f084099d4b37b67aba3cda9ecff3', 'request_id': 'req-8449d620-d936-41ae-8779-263a1798eb0a'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:03.929248 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-8449d620-d936-41ae-8779-263a1798eb0a admin admin] The token cache is being invalidate because project d5f8f084099d4b37b67aba3cda9ecff3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:58:03.929566 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-8449d620-d936-41ae-8779-263a1798eb0a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:03.933627 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-8449d620-d936-41ae-8779-263a1798eb0a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:03.938080 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1143/2276] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:58:03 2022] DELETE /identity/v3/projects/d5f8f084099d4b37b67aba3cda9ecff3 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:58:06.625781 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0f61aa-e018-4106-aaec-87b031133222 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:06.626292 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0f61aa-e018-4106-aaec-87b031133222 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:06.626377 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0f61aa-e018-4106-aaec-87b031133222 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:06.663119 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-cf0f61aa-e018-4106-aaec-87b031133222 None None] MFA Rules not processed for user `80dd52f1455f44b587716a41e632a155`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:58:06.664512 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-cf0f61aa-e018-4106-aaec-87b031133222 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:06.746496 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1134/2277] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:58:06 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:58:06.751383 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8449d620-d936-41ae-8779-263a1798eb0a admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:06.753248 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-8449d620-d936-41ae-8779-263a1798eb0a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:06.830803 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8449d620-d936-41ae-8779-263a1798eb0a admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:06.836816 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a54423c-be0f-4d4a-afd6-a867d95295fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:06.837447 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a54423c-be0f-4d4a-afd6-a867d95295fb admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:06.837865 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a54423c-be0f-4d4a-afd6-a867d95295fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:06.838146 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a54423c-be0f-4d4a-afd6-a867d95295fb admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:06.839340 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a54423c-be0f-4d4a-afd6-a867d95295fb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:06.840940 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a54423c-be0f-4d4a-afd6-a867d95295fb admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:06.887850 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1144/2278] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:58:06 2022] POST /identity/v3/projects => generated 385 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:58:06.888537 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf0f61aa-e018-4106-aaec-87b031133222 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:06.900572 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf0f61aa-e018-4106-aaec-87b031133222 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:06.987830 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed11b5ae-6fa3-4b37-bbc3-e5615e5459ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:06.988457 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed11b5ae-6fa3-4b37-bbc3-e5615e5459ad admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:06.988730 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed11b5ae-6fa3-4b37-bbc3-e5615e5459ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:06.989022 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed11b5ae-6fa3-4b37-bbc3-e5615e5459ad admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:06.989896 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed11b5ae-6fa3-4b37-bbc3-e5615e5459ad admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:06.992176 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed11b5ae-6fa3-4b37-bbc3-e5615e5459ad admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:07.025792 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1135/2279] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:58:06 2022] POST /identity/v3/users => generated 299 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:58:07.027008 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a54423c-be0f-4d4a-afd6-a867d95295fb admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:07.042339 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a54423c-be0f-4d4a-afd6-a867d95295fb admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:07.047898 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93469290-5e6e-424a-98b1-98dbfbb1135a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:07.048539 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93469290-5e6e-424a-98b1-98dbfbb1135a admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:07.048950 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93469290-5e6e-424a-98b1-98dbfbb1135a admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:07.049343 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93469290-5e6e-424a-98b1-98dbfbb1135a admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:07.050214 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93469290-5e6e-424a-98b1-98dbfbb1135a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:07.052295 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93469290-5e6e-424a-98b1-98dbfbb1135a admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:07.067570 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1145/2280] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:58:07 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 00:58:07.071769 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed11b5ae-6fa3-4b37-bbc3-e5615e5459ad admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:07.084762 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed11b5ae-6fa3-4b37-bbc3-e5615e5459ad admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:07.091500 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c83fdf49-ab95-4f85-8cd3-c15a23f10c5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:07.092268 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83fdf49-ab95-4f85-8cd3-c15a23f10c5d admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:07.092562 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83fdf49-ab95-4f85-8cd3-c15a23f10c5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:07.092864 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83fdf49-ab95-4f85-8cd3-c15a23f10c5d admin admin] PATH_INFO: `/v3/projects/31719d4e28f048bbbf6e307df745e648/users/972219920bf74507b005a7c9d10dd6bd/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:07.113306 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c83fdf49-ab95-4f85-8cd3-c15a23f10c5d admin admin] RBAC: Authorizing `identity:create_grant(project_id=31719d4e28f048bbbf6e307df745e648, user_id=972219920bf74507b005a7c9d10dd6bd, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:07.116843 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c83fdf49-ab95-4f85-8cd3-c15a23f10c5d admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:07.132126 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1136/2281] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:58:07 2022] PUT /identity/v3/projects/31719d4e28f048bbbf6e307df745e648/users/972219920bf74507b005a7c9d10dd6bd/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:58:07.136223 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93469290-5e6e-424a-98b1-98dbfbb1135a admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:07.147807 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93469290-5e6e-424a-98b1-98dbfbb1135a admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:07.261830 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:58:07.293951 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09050bad-d88d-48ed-b48e-860ecd4eb54a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:07.294785 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09050bad-d88d-48ed-b48e-860ecd4eb54a admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:07.295073 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09050bad-d88d-48ed-b48e-860ecd4eb54a admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:07.295576 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09050bad-d88d-48ed-b48e-860ecd4eb54a admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:07.296296 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09050bad-d88d-48ed-b48e-860ecd4eb54a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:07.298172 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09050bad-d88d-48ed-b48e-860ecd4eb54a admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:07.312400 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1146/2282] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:58:07 2022] GET /identity/v3/roles => generated 1420 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:58:07.316975 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c83fdf49-ab95-4f85-8cd3-c15a23f10c5d admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:07.330276 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c83fdf49-ab95-4f85-8cd3-c15a23f10c5d admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:07.337878 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70ff3fd-c0d7-43f0-addd-6e8b6c4f17c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:07.338557 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70ff3fd-c0d7-43f0-addd-6e8b6c4f17c4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:07.338833 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70ff3fd-c0d7-43f0-addd-6e8b6c4f17c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:07.339160 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70ff3fd-c0d7-43f0-addd-6e8b6c4f17c4 admin admin] PATH_INFO: `/v3/projects/31719d4e28f048bbbf6e307df745e648/users/972219920bf74507b005a7c9d10dd6bd/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:07.342614 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70ff3fd-c0d7-43f0-addd-6e8b6c4f17c4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=31719d4e28f048bbbf6e307df745e648, user_id=972219920bf74507b005a7c9d10dd6bd, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:07.345322 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70ff3fd-c0d7-43f0-addd-6e8b6c4f17c4 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:07.349106 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:58:07.363251 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1137/2283] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:58:07 2022] PUT /identity/v3/projects/31719d4e28f048bbbf6e307df745e648/users/972219920bf74507b005a7c9d10dd6bd/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 43 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:58:08.682992 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3880f82-f88c-4d7a-b067-b6ee2c59fc0c None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:08.684265 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3880f82-f88c-4d7a-b067-b6ee2c59fc0c None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:08.685370 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3880f82-f88c-4d7a-b067-b6ee2c59fc0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:08.733464 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-a3880f82-f88c-4d7a-b067-b6ee2c59fc0c None None] MFA Rules not processed for user `96386a16f0b54b7c94d32bd948caf118`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:58:08.734720 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-a3880f82-f88c-4d7a-b067-b6ee2c59fc0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:08.903584 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1147/2284] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:58:08 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 221 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:58:10.096616 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87440df-2264-4449-b702-f876ff306f3e None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:10.097180 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87440df-2264-4449-b702-f876ff306f3e None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:10.097491 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87440df-2264-4449-b702-f876ff306f3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:10.154184 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-f87440df-2264-4449-b702-f876ff306f3e None None] MFA Rules not processed for user `c19b10ebc31244dbb2b1539d94ac281d`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:58:10.158343 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-f87440df-2264-4449-b702-f876ff306f3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:10.310338 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1138/2285] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:58:10 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:58:14.460246 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b1ed16-ae81-4f51-8cdb-e379e9f627aa None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:14.461132 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b1ed16-ae81-4f51-8cdb-e379e9f627aa None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:14.461598 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b1ed16-ae81-4f51-8cdb-e379e9f627aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:14.501994 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-08b1ed16-ae81-4f51-8cdb-e379e9f627aa None None] MFA Rules not processed for user `972219920bf74507b005a7c9d10dd6bd`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:58:14.504301 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-08b1ed16-ae81-4f51-8cdb-e379e9f627aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:14.601635 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1148/2286] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:58:14 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:58:14.610568 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f87440df-2264-4449-b702-f876ff306f3e None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:14.612594 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-f87440df-2264-4449-b702-f876ff306f3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:14.698964 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f87440df-2264-4449-b702-f876ff306f3e None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:14.702819 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8942f12e-6418-4bb4-bb20-ef082b5c316c service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4ebfc0abf1b842459a44ca9324252688', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:14.703543 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8942f12e-6418-4bb4-bb20-ef082b5c316c service glance] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:14.703992 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8942f12e-6418-4bb4-bb20-ef082b5c316c service glance] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:14.704264 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8942f12e-6418-4bb4-bb20-ef082b5c316c service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:14.779888 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8942f12e-6418-4bb4-bb20-ef082b5c316c service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:14.781381 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8942f12e-6418-4bb4-bb20-ef082b5c316c service glance] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:14.791658 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1139/2287] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:58:14 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 182 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 00:58:14.998683 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc994fb0-d3e9-49f2-8c15-5fa65b26fa56 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:14.999266 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc994fb0-d3e9-49f2-8c15-5fa65b26fa56 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:14.999644 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc994fb0-d3e9-49f2-8c15-5fa65b26fa56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:15.025583 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-dc994fb0-d3e9-49f2-8c15-5fa65b26fa56 None None] MFA Rules not processed for user `8eabe2a74a604c47bc2e5c88b960dcb6`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:58:15.026813 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-dc994fb0-d3e9-49f2-8c15-5fa65b26fa56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:15.027035 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c276b025-0866-4c16-8c1e-eb9765798df8 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:15.027314 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c276b025-0866-4c16-8c1e-eb9765798df8 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:15.027578 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c276b025-0866-4c16-8c1e-eb9765798df8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:15.052430 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-c276b025-0866-4c16-8c1e-eb9765798df8 None None] MFA Rules not processed for user `3f20ffd8df994c1db1b1e7248db5b5a3`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:58:15.053741 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-c276b025-0866-4c16-8c1e-eb9765798df8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:15.097431 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1149/2288] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:58:14 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:58:15.115767 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:58:15.126116 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1140/2289] 10.222.0.52 () {58 vars in 1009 bytes} [Sun Sep 25 00:58:15 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 00:58:15.181198 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc994fb0-d3e9-49f2-8c15-5fa65b26fa56 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:15.182488 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-dc994fb0-d3e9-49f2-8c15-5fa65b26fa56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:15.195893 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:58:15.334508 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc994fb0-d3e9-49f2-8c15-5fa65b26fa56 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:15.341218 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43a9803-23c0-4f0b-a557-5e103a15f5aa service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da606b3906dd4171bd76fd281e03e258', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:15.342623 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43a9803-23c0-4f0b-a557-5e103a15f5aa service swift] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:15.342905 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43a9803-23c0-4f0b-a557-5e103a15f5aa service swift] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:15.343182 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43a9803-23c0-4f0b-a557-5e103a15f5aa service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:15.353780 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f43a9803-23c0-4f0b-a557-5e103a15f5aa service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:15.355474 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f43a9803-23c0-4f0b-a557-5e103a15f5aa service swift] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:15.374235 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1150/2290] 10.222.0.52 () {60 vars in 1384 bytes} [Sun Sep 25 00:58:15 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 194 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 25 00:58:25.361977 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=28 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:58:30.548054 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c276b025-0866-4c16-8c1e-eb9765798df8 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:30.552435 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-c276b025-0866-4c16-8c1e-eb9765798df8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:30.669114 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c276b025-0866-4c16-8c1e-eb9765798df8 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:30.674144 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-298d31ed-ed55-43b3-a45f-0ab352b4db4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:30.674978 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298d31ed-ed55-43b3-a45f-0ab352b4db4e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:30.675404 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298d31ed-ed55-43b3-a45f-0ab352b4db4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:30.676238 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298d31ed-ed55-43b3-a45f-0ab352b4db4e admin admin] PATH_INFO: `/v3/users/b46132eeb4104edf88474e9f80e2405d` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:30.677663 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-298d31ed-ed55-43b3-a45f-0ab352b4db4e admin admin] RBAC: Authorizing `identity:delete_user(user_id=b46132eeb4104edf88474e9f80e2405d)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:30.679599 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-298d31ed-ed55-43b3-a45f-0ab352b4db4e admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:30.739211 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-298d31ed-ed55-43b3-a45f-0ab352b4db4e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b46132eeb4104edf88474e9f80e2405d'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:30.748087 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-298d31ed-ed55-43b3-a45f-0ab352b4db4e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b46132eeb4104edf88474e9f80e2405d'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:30.748964 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-298d31ed-ed55-43b3-a45f-0ab352b4db4e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b46132eeb4104edf88474e9f80e2405d'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:30.757693 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-298d31ed-ed55-43b3-a45f-0ab352b4db4e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b46132eeb4104edf88474e9f80e2405d'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:30.758401 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-298d31ed-ed55-43b3-a45f-0ab352b4db4e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b46132eeb4104edf88474e9f80e2405d'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:30.780639 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1141/2291] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:58:30 2022] DELETE /identity/v3/users/b46132eeb4104edf88474e9f80e2405d => generated 0 bytes in 233 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:58:30.786094 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a8b431-ca16-4920-b9c6-3a997adda998 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:30.786601 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a8b431-ca16-4920-b9c6-3a997adda998 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:30.786682 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a8b431-ca16-4920-b9c6-3a997adda998 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:30.790941 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0307ea9-49d3-401e-a01f-64d861f277dd None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:30.791229 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0307ea9-49d3-401e-a01f-64d861f277dd None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:30.791503 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0307ea9-49d3-401e-a01f-64d861f277dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:30.806908 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-37a8b431-ca16-4920-b9c6-3a997adda998 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:58:30.808194 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-37a8b431-ca16-4920-b9c6-3a997adda998 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:30.813242 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-b0307ea9-49d3-401e-a01f-64d861f277dd None None] MFA Rules not processed for user `80dd52f1455f44b587716a41e632a155`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:58:30.814719 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-b0307ea9-49d3-401e-a01f-64d861f277dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:30.878345 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1151/2292] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:58:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:58:30.885576 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1142/2293] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:58:30 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:58:30.894390 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a8b431-ca16-4920-b9c6-3a997adda998 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:30.896019 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-37a8b431-ca16-4920-b9c6-3a997adda998 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:30.955463 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a8b431-ca16-4920-b9c6-3a997adda998 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:30.959504 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdebc94c-c1c4-468f-8edd-285988c9d6a9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:30.960036 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdebc94c-c1c4-468f-8edd-285988c9d6a9 service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:30.960305 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdebc94c-c1c4-468f-8edd-285988c9d6a9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:30.960708 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdebc94c-c1c4-468f-8edd-285988c9d6a9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:30.970481 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdebc94c-c1c4-468f-8edd-285988c9d6a9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:30.972207 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdebc94c-c1c4-468f-8edd-285988c9d6a9 service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:30.981827 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1152/2294] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:58:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:58:31.524934 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0307ea9-49d3-401e-a01f-64d861f277dd None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:31.526367 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-b0307ea9-49d3-401e-a01f-64d861f277dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:31.547767 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0307ea9-49d3-401e-a01f-64d861f277dd None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:31.553669 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29a58d5-a096-4165-ac97-ee6fef9c0c88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:31.555746 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29a58d5-a096-4165-ac97-ee6fef9c0c88 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:31.556468 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29a58d5-a096-4165-ac97-ee6fef9c0c88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:31.556882 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29a58d5-a096-4165-ac97-ee6fef9c0c88 admin admin] PATH_INFO: `/v3/projects/3880cc1b631a4426adc5cae65932049d` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:31.558255 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d29a58d5-a096-4165-ac97-ee6fef9c0c88 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3880cc1b631a4426adc5cae65932049d)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:31.560312 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d29a58d5-a096-4165-ac97-ee6fef9c0c88 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:31.599758 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:58:31.648172 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-d29a58d5-a096-4165-ac97-ee6fef9c0c88 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3880cc1b631a4426adc5cae65932049d', 'request_id': 'req-d29a58d5-a096-4165-ac97-ee6fef9c0c88'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:31.660715 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-d29a58d5-a096-4165-ac97-ee6fef9c0c88 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3880cc1b631a4426adc5cae65932049d', 'request_id': 'req-d29a58d5-a096-4165-ac97-ee6fef9c0c88'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:31.667903 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-d29a58d5-a096-4165-ac97-ee6fef9c0c88 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3880cc1b631a4426adc5cae65932049d', 'request_id': 'req-d29a58d5-a096-4165-ac97-ee6fef9c0c88'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:31.682602 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-d29a58d5-a096-4165-ac97-ee6fef9c0c88 admin admin] The token cache is being invalidate because project 3880cc1b631a4426adc5cae65932049d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81386) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:58:31.682985 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-d29a58d5-a096-4165-ac97-ee6fef9c0c88 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:31.683734 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-d29a58d5-a096-4165-ac97-ee6fef9c0c88 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:31.688390 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1143/2295] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:58:31 2022] DELETE /identity/v3/projects/3880cc1b631a4426adc5cae65932049d => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:58:31.689918 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdebc94c-c1c4-468f-8edd-285988c9d6a9 service neutron] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:31.721921 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdebc94c-c1c4-468f-8edd-285988c9d6a9 service neutron] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:31.874918 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a573a5a4-9ae5-4bc6-8c7d-b26e70eca313 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:31.876114 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a573a5a4-9ae5-4bc6-8c7d-b26e70eca313 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:31.876386 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a573a5a4-9ae5-4bc6-8c7d-b26e70eca313 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:31.876654 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a573a5a4-9ae5-4bc6-8c7d-b26e70eca313 admin admin] PATH_INFO: `/v3/users/6109db02b8a6424da5c67098cd8806bb` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:31.880955 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a573a5a4-9ae5-4bc6-8c7d-b26e70eca313 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6109db02b8a6424da5c67098cd8806bb)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:31.883095 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a573a5a4-9ae5-4bc6-8c7d-b26e70eca313 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:31.997319 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:58:31.999643 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-a573a5a4-9ae5-4bc6-8c7d-b26e70eca313 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6109db02b8a6424da5c67098cd8806bb'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:32.000633 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-a573a5a4-9ae5-4bc6-8c7d-b26e70eca313 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6109db02b8a6424da5c67098cd8806bb'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:32.014682 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-a573a5a4-9ae5-4bc6-8c7d-b26e70eca313 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6109db02b8a6424da5c67098cd8806bb'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:32.015299 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-a573a5a4-9ae5-4bc6-8c7d-b26e70eca313 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6109db02b8a6424da5c67098cd8806bb'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:32.027439 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-a573a5a4-9ae5-4bc6-8c7d-b26e70eca313 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6109db02b8a6424da5c67098cd8806bb'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:32.051621 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1153/2296] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:58:31 2022] DELETE /identity/v3/users/6109db02b8a6424da5c67098cd8806bb => generated 0 bytes in 362 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:58:32.423542 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29a58d5-a096-4165-ac97-ee6fef9c0c88 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:32.425856 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-d29a58d5-a096-4165-ac97-ee6fef9c0c88 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:32.440362 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29a58d5-a096-4165-ac97-ee6fef9c0c88 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:32.517084 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d393c6c6-89b4-44f1-8c27-75dfc12f5f6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:32.517833 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d393c6c6-89b4-44f1-8c27-75dfc12f5f6c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:32.518110 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d393c6c6-89b4-44f1-8c27-75dfc12f5f6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:32.518382 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d393c6c6-89b4-44f1-8c27-75dfc12f5f6c admin admin] PATH_INFO: `/v3/projects/be6239167c474888803de4aa4039c40e` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:32.519586 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d393c6c6-89b4-44f1-8c27-75dfc12f5f6c admin admin] RBAC: Authorizing `identity:delete_project(project_id=be6239167c474888803de4aa4039c40e)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:32.521601 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d393c6c6-89b4-44f1-8c27-75dfc12f5f6c admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:32.604462 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-d393c6c6-89b4-44f1-8c27-75dfc12f5f6c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'be6239167c474888803de4aa4039c40e', 'request_id': 'req-d393c6c6-89b4-44f1-8c27-75dfc12f5f6c'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:32.615102 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-d393c6c6-89b4-44f1-8c27-75dfc12f5f6c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'be6239167c474888803de4aa4039c40e', 'request_id': 'req-d393c6c6-89b4-44f1-8c27-75dfc12f5f6c'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:32.642052 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-d393c6c6-89b4-44f1-8c27-75dfc12f5f6c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'be6239167c474888803de4aa4039c40e', 'request_id': 'req-d393c6c6-89b4-44f1-8c27-75dfc12f5f6c'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:32.667721 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-d393c6c6-89b4-44f1-8c27-75dfc12f5f6c admin admin] The token cache is being invalidate because project be6239167c474888803de4aa4039c40e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81386) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:58:32.668231 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-d393c6c6-89b4-44f1-8c27-75dfc12f5f6c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:32.669228 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-d393c6c6-89b4-44f1-8c27-75dfc12f5f6c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:32.671354 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1144/2297] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:58:32 2022] DELETE /identity/v3/projects/be6239167c474888803de4aa4039c40e => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:58:32.677923 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bae98de-78bb-4771-a232-cd70851eb9bb None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:32.678585 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bae98de-78bb-4771-a232-cd70851eb9bb None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:32.679112 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bae98de-78bb-4771-a232-cd70851eb9bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:32.709480 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-6bae98de-78bb-4771-a232-cd70851eb9bb None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:58:32.713376 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-6bae98de-78bb-4771-a232-cd70851eb9bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:32.731878 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393d7b3d-9f31-4174-90ea-e719ee9f9bde None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:32.732491 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393d7b3d-9f31-4174-90ea-e719ee9f9bde None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:32.733344 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393d7b3d-9f31-4174-90ea-e719ee9f9bde None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:32.761457 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-393d7b3d-9f31-4174-90ea-e719ee9f9bde None None] MFA Rules not processed for user `3f20ffd8df994c1db1b1e7248db5b5a3`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:58:32.763226 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-393d7b3d-9f31-4174-90ea-e719ee9f9bde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:32.806786 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1154/2298] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:58:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:58:32.814975 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a957a76c-35a0-4392-810b-ed126023b78b None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:32.815399 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a957a76c-35a0-4392-810b-ed126023b78b None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:32.815770 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a957a76c-35a0-4392-810b-ed126023b78b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:32.851238 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-a957a76c-35a0-4392-810b-ed126023b78b None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:58:32.867947 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-a957a76c-35a0-4392-810b-ed126023b78b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:32.868454 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:58:32.874382 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1155/2299] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:58:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 60 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 00:58:32.879504 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a957a76c-35a0-4392-810b-ed126023b78b None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:32.894119 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1145/2300] 10.222.0.52 () {58 vars in 1009 bytes} [Sun Sep 25 00:58:32 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:58:32.894978 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a957a76c-35a0-4392-810b-ed126023b78b None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:32.907111 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-393d7b3d-9f31-4174-90ea-e719ee9f9bde None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:32.908564 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-393d7b3d-9f31-4174-90ea-e719ee9f9bde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:32.925397 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04eeffbc-a693-4e35-b69c-b3ba1bfc8705 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:32.927025 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04eeffbc-a693-4e35-b69c-b3ba1bfc8705 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:32.927309 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04eeffbc-a693-4e35-b69c-b3ba1bfc8705 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:32.928333 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04eeffbc-a693-4e35-b69c-b3ba1bfc8705 admin admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:32.929009 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04eeffbc-a693-4e35-b69c-b3ba1bfc8705 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:58:32.929334 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04eeffbc-a693-4e35-b69c-b3ba1bfc8705 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:32.936877 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04eeffbc-a693-4e35-b69c-b3ba1bfc8705 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:32.961564 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1156/2301] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 00:58:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:58:32.965393 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04eeffbc-a693-4e35-b69c-b3ba1bfc8705 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:32.988071 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04eeffbc-a693-4e35-b69c-b3ba1bfc8705 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:33.002284 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3c34c3-bb02-4ce4-8da7-f2c2e02c404d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:33.004016 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3c34c3-bb02-4ce4-8da7-f2c2e02c404d admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:33.004483 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3c34c3-bb02-4ce4-8da7-f2c2e02c404d admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:33.005036 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3c34c3-bb02-4ce4-8da7-f2c2e02c404d admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:33.006248 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff3c34c3-bb02-4ce4-8da7-f2c2e02c404d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:33.008419 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff3c34c3-bb02-4ce4-8da7-f2c2e02c404d admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:33.050110 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1157/2302] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:58:32 2022] POST /identity/v3/projects => generated 357 bytes in 85 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:58:33.050379 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-393d7b3d-9f31-4174-90ea-e719ee9f9bde None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:33.062302 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3c34c3-bb02-4ce4-8da7-f2c2e02c404d admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:33.161080 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-063321ef-7576-4708-bcb7-ec4f76db2613 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da606b3906dd4171bd76fd281e03e258', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:33.162006 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063321ef-7576-4708-bcb7-ec4f76db2613 service swift] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:33.162373 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063321ef-7576-4708-bcb7-ec4f76db2613 service swift] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:33.162809 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063321ef-7576-4708-bcb7-ec4f76db2613 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:33.186048 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3c34c3-bb02-4ce4-8da7-f2c2e02c404d admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:33.192724 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d347b4-7e42-4e82-8ed4-807ed0e48c14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:33.193461 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d347b4-7e42-4e82-8ed4-807ed0e48c14 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:33.193944 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d347b4-7e42-4e82-8ed4-807ed0e48c14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:33.194434 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d347b4-7e42-4e82-8ed4-807ed0e48c14 admin admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:33.195165 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3d347b4-7e42-4e82-8ed4-807ed0e48c14 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:33.197410 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3d347b4-7e42-4e82-8ed4-807ed0e48c14 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:33.234206 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1158/2303] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:58:33 2022] POST /identity/v3/users => generated 279 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:58:33.235399 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d347b4-7e42-4e82-8ed4-807ed0e48c14 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:33.252335 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d347b4-7e42-4e82-8ed4-807ed0e48c14 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:33.258508 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7deab39c-9000-42a0-b750-130d94d8b857 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:33.259428 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7deab39c-9000-42a0-b750-130d94d8b857 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:33.259428 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7deab39c-9000-42a0-b750-130d94d8b857 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:33.259756 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7deab39c-9000-42a0-b750-130d94d8b857 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:33.260298 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7deab39c-9000-42a0-b750-130d94d8b857 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:33.262543 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7deab39c-9000-42a0-b750-130d94d8b857 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:33.263490 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-063321ef-7576-4708-bcb7-ec4f76db2613 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:33.266080 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-063321ef-7576-4708-bcb7-ec4f76db2613 service swift] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:33.269813 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=92,keystone:INSERT=5,keystone:DELETE=3 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:58:33.277434 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1159/2304] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:58:33 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:58:33.281467 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7deab39c-9000-42a0-b750-130d94d8b857 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:33.285460 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1146/2305] 10.222.0.52 () {60 vars in 1384 bytes} [Sun Sep 25 00:58:32 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 25 00:58:33.293983 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7deab39c-9000-42a0-b750-130d94d8b857 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:33.300071 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eceffecb-0563-471e-8002-82c49c7fe125 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:33.300769 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eceffecb-0563-471e-8002-82c49c7fe125 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:33.301213 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eceffecb-0563-471e-8002-82c49c7fe125 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:33.301634 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eceffecb-0563-471e-8002-82c49c7fe125 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:33.302434 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eceffecb-0563-471e-8002-82c49c7fe125 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:33.304346 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eceffecb-0563-471e-8002-82c49c7fe125 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:33.322644 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1160/2306] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:58:33 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:58:33.323910 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-063321ef-7576-4708-bcb7-ec4f76db2613 service swift] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:33.342307 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-063321ef-7576-4708-bcb7-ec4f76db2613 service swift] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:33.348704 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7db198c-582e-42d5-9f59-1b57bc4011ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:33.349675 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7db198c-582e-42d5-9f59-1b57bc4011ef admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:33.350045 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7db198c-582e-42d5-9f59-1b57bc4011ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:33.350418 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7db198c-582e-42d5-9f59-1b57bc4011ef admin admin] PATH_INFO: `/v3/projects/fe51531385464725be3c70456b493e19/users/f705965312b541bd9e9168ea44b328b2/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:33.374880 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7db198c-582e-42d5-9f59-1b57bc4011ef admin admin] RBAC: Authorizing `identity:create_grant(project_id=fe51531385464725be3c70456b493e19, user_id=f705965312b541bd9e9168ea44b328b2, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:33.379259 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7db198c-582e-42d5-9f59-1b57bc4011ef admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:33.405029 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1147/2307] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:58:33 2022] PUT /identity/v3/projects/fe51531385464725be3c70456b493e19/users/f705965312b541bd9e9168ea44b328b2/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 82 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 00:58:33.416380 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fcb18bd-4c35-44ce-b6ce-c3c2fee47924 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:33.416746 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fcb18bd-4c35-44ce-b6ce-c3c2fee47924 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:33.417049 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fcb18bd-4c35-44ce-b6ce-c3c2fee47924 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:33.459370 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-0fcb18bd-4c35-44ce-b6ce-c3c2fee47924 None None] MFA Rules not processed for user `f705965312b541bd9e9168ea44b328b2`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:58:33.460623 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-0fcb18bd-4c35-44ce-b6ce-c3c2fee47924 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:33.556103 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1161/2308] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:58:33 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:58:33.560668 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7db198c-582e-42d5-9f59-1b57bc4011ef admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:33.566074 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:58:33.644737 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7db198c-582e-42d5-9f59-1b57bc4011ef admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:33.649971 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f5c3751-a0f9-4732-aa42-0abaf6d0a849 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:33.652204 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5c3751-a0f9-4732-aa42-0abaf6d0a849 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:33.654168 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5c3751-a0f9-4732-aa42-0abaf6d0a849 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:33.654673 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5c3751-a0f9-4732-aa42-0abaf6d0a849 admin admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:33.655701 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f5c3751-a0f9-4732-aa42-0abaf6d0a849 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:33.659288 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f5c3751-a0f9-4732-aa42-0abaf6d0a849 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:33.703034 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1148/2309] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:58:33 2022] POST /identity/v3/projects => generated 357 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 00:58:33.705493 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fcb18bd-4c35-44ce-b6ce-c3c2fee47924 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:33.725348 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fcb18bd-4c35-44ce-b6ce-c3c2fee47924 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:33.804648 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68ca988-d3f5-47f1-8b29-a8bfd8b8f9d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:33.805606 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68ca988-d3f5-47f1-8b29-a8bfd8b8f9d5 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:33.806035 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68ca988-d3f5-47f1-8b29-a8bfd8b8f9d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:33.806498 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68ca988-d3f5-47f1-8b29-a8bfd8b8f9d5 admin admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:33.807318 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b68ca988-d3f5-47f1-8b29-a8bfd8b8f9d5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:33.809290 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b68ca988-d3f5-47f1-8b29-a8bfd8b8f9d5 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:33.847053 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1162/2310] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:58:33 2022] POST /identity/v3/users => generated 285 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:58:33.848666 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f5c3751-a0f9-4732-aa42-0abaf6d0a849 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:33.863909 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f5c3751-a0f9-4732-aa42-0abaf6d0a849 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:33.874390 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea285844-60fc-4c1a-b8be-029e3239ef0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:33.875038 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea285844-60fc-4c1a-b8be-029e3239ef0f admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:33.875452 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea285844-60fc-4c1a-b8be-029e3239ef0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:33.875562 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea285844-60fc-4c1a-b8be-029e3239ef0f admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:33.877654 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea285844-60fc-4c1a-b8be-029e3239ef0f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:33.879288 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea285844-60fc-4c1a-b8be-029e3239ef0f admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:33.910824 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1149/2311] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:58:33 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:58:33.914193 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68ca988-d3f5-47f1-8b29-a8bfd8b8f9d5 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:33.931673 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68ca988-d3f5-47f1-8b29-a8bfd8b8f9d5 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:33.944263 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-623eddcf-24ff-4871-82e9-3391af008678 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:33.944988 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623eddcf-24ff-4871-82e9-3391af008678 admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:33.945282 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623eddcf-24ff-4871-82e9-3391af008678 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:33.945616 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623eddcf-24ff-4871-82e9-3391af008678 admin admin] PATH_INFO: `/v3/projects/5883265470f0434a87df7adc31ceff1e/users/aab4adc8b502477494b3b0d802a470eb/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:33.989072 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-623eddcf-24ff-4871-82e9-3391af008678 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5883265470f0434a87df7adc31ceff1e, user_id=aab4adc8b502477494b3b0d802a470eb, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:33.997643 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-623eddcf-24ff-4871-82e9-3391af008678 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:34.027991 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1163/2312] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:58:33 2022] PUT /identity/v3/projects/5883265470f0434a87df7adc31ceff1e/users/aab4adc8b502477494b3b0d802a470eb/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:58:34.031714 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea285844-60fc-4c1a-b8be-029e3239ef0f admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:34.046297 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea285844-60fc-4c1a-b8be-029e3239ef0f admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:34.227037 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f32d78b6-f05a-4051-bd7e-7f2f89da4da7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:34.227647 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f32d78b6-f05a-4051-bd7e-7f2f89da4da7 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:34.227978 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f32d78b6-f05a-4051-bd7e-7f2f89da4da7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:34.228245 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f32d78b6-f05a-4051-bd7e-7f2f89da4da7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:34.228891 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f32d78b6-f05a-4051-bd7e-7f2f89da4da7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:34.230527 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f32d78b6-f05a-4051-bd7e-7f2f89da4da7 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:34.255306 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1150/2313] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:58:34 2022] GET /identity/v3/roles => generated 1420 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:58:34.260235 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-623eddcf-24ff-4871-82e9-3391af008678 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:34.273006 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-623eddcf-24ff-4871-82e9-3391af008678 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:34.278876 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1afdc0e3-e3fb-42d4-9efe-04e5e770ebda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:34.279710 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afdc0e3-e3fb-42d4-9efe-04e5e770ebda admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:34.279987 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afdc0e3-e3fb-42d4-9efe-04e5e770ebda admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:34.280251 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afdc0e3-e3fb-42d4-9efe-04e5e770ebda admin admin] PATH_INFO: `/v3/projects/5883265470f0434a87df7adc31ceff1e/users/aab4adc8b502477494b3b0d802a470eb/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:34.282443 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1afdc0e3-e3fb-42d4-9efe-04e5e770ebda admin admin] RBAC: Authorizing `identity:create_grant(project_id=5883265470f0434a87df7adc31ceff1e, user_id=aab4adc8b502477494b3b0d802a470eb, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:34.288166 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1afdc0e3-e3fb-42d4-9efe-04e5e770ebda admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:34.304573 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1164/2314] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:58:34 2022] PUT /identity/v3/projects/5883265470f0434a87df7adc31ceff1e/users/aab4adc8b502477494b3b0d802a470eb/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 45 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:58:34.315223 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60f6041-6e36-4983-90db-4c9d5eb380c5 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:34.315721 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60f6041-6e36-4983-90db-4c9d5eb380c5 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:34.316710 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60f6041-6e36-4983-90db-4c9d5eb380c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:34.359101 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-b60f6041-6e36-4983-90db-4c9d5eb380c5 None None] MFA Rules not processed for user `aab4adc8b502477494b3b0d802a470eb`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:58:34.360962 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-b60f6041-6e36-4983-90db-4c9d5eb380c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:34.486980 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:58:34.492075 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1151/2315] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:58:34 2022] POST /identity/v3/auth/tokens => generated 3301 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:58:34.499736 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1afdc0e3-e3fb-42d4-9efe-04e5e770ebda admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:34.503070 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-1afdc0e3-e3fb-42d4-9efe-04e5e770ebda admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:34.571682 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:58:34.609438 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1afdc0e3-e3fb-42d4-9efe-04e5e770ebda admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:34.614205 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-862b78db-e8b6-4223-bcc8-a76673995df3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:34.614913 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862b78db-e8b6-4223-bcc8-a76673995df3 service cinder] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:34.615264 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862b78db-e8b6-4223-bcc8-a76673995df3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:34.615601 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862b78db-e8b6-4223-bcc8-a76673995df3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:34.627517 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-862b78db-e8b6-4223-bcc8-a76673995df3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:34.630510 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-862b78db-e8b6-4223-bcc8-a76673995df3 service cinder] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:34.675242 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1165/2316] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:58:34 2022] GET /identity/v3/auth/tokens => generated 3301 bytes in 170 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:58:34.789423 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60f6041-6e36-4983-90db-4c9d5eb380c5 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:34.801540 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60f6041-6e36-4983-90db-4c9d5eb380c5 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:34.813748 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e778ed2-0833-42ed-b84e-875344e86613 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:34.815090 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e778ed2-0833-42ed-b84e-875344e86613 service cinder] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:34.815470 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e778ed2-0833-42ed-b84e-875344e86613 service cinder] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:34.815838 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e778ed2-0833-42ed-b84e-875344e86613 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:34.893227 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e778ed2-0833-42ed-b84e-875344e86613 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:34.895192 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e778ed2-0833-42ed-b84e-875344e86613 service cinder] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:34.906798 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1152/2317] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:58:34 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:58:35.310534 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c36f72-a99f-498d-8a29-4eca971d8d37 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:35.311129 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c36f72-a99f-498d-8a29-4eca971d8d37 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:35.311129 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c36f72-a99f-498d-8a29-4eca971d8d37 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:35.351243 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-11c36f72-a99f-498d-8a29-4eca971d8d37 None None] MFA Rules not processed for user `3f20ffd8df994c1db1b1e7248db5b5a3`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:58:35.352989 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-11c36f72-a99f-498d-8a29-4eca971d8d37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:35.527544 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1166/2318] 10.222.0.52 () {58 vars in 1009 bytes} [Sun Sep 25 00:58:35 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 232 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:58:35.535618 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e778ed2-0833-42ed-b84e-875344e86613 service cinder] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:35.692395 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e778ed2-0833-42ed-b84e-875344e86613 service cinder] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:35.698720 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c22d26-0c0a-4d6a-8c43-59a8a651d9e2 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da606b3906dd4171bd76fd281e03e258', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:35.699344 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c22d26-0c0a-4d6a-8c43-59a8a651d9e2 service swift] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:35.699620 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c22d26-0c0a-4d6a-8c43-59a8a651d9e2 service swift] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:35.699888 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c22d26-0c0a-4d6a-8c43-59a8a651d9e2 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:35.710049 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7c22d26-0c0a-4d6a-8c43-59a8a651d9e2 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:35.711824 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7c22d26-0c0a-4d6a-8c43-59a8a651d9e2 service swift] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:35.726591 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1153/2319] 10.222.0.52 () {60 vars in 1384 bytes} [Sun Sep 25 00:58:35 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 191 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 25 00:58:45.490032 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=37 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:58:45.717327 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=61 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:58:48.645679 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11c36f72-a99f-498d-8a29-4eca971d8d37 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:48.652951 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-11c36f72-a99f-498d-8a29-4eca971d8d37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:48.759979 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11c36f72-a99f-498d-8a29-4eca971d8d37 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:48.772497 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaedca11-4e15-4ee3-88bb-234836924dd6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:48.773532 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaedca11-4e15-4ee3-88bb-234836924dd6 service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:48.774059 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaedca11-4e15-4ee3-88bb-234836924dd6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:48.774812 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaedca11-4e15-4ee3-88bb-234836924dd6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:48.777084 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-eaedca11-4e15-4ee3-88bb-234836924dd6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:48.897389 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaedca11-4e15-4ee3-88bb-234836924dd6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:48.897389 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaedca11-4e15-4ee3-88bb-234836924dd6 service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:48.904772 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1167/2320] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:58:48 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 261 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:58:54.275044 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c22d26-0c0a-4d6a-8c43-59a8a651d9e2 service swift] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:54.385838 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c22d26-0c0a-4d6a-8c43-59a8a651d9e2 service swift] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:54.394931 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bfff737-4fbb-4881-addc-dad872089e60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:54.395581 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfff737-4fbb-4881-addc-dad872089e60 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:54.395852 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfff737-4fbb-4881-addc-dad872089e60 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:54.396115 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfff737-4fbb-4881-addc-dad872089e60 admin admin] PATH_INFO: `/v3/users/f705965312b541bd9e9168ea44b328b2` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:54.397331 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bfff737-4fbb-4881-addc-dad872089e60 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f705965312b541bd9e9168ea44b328b2)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:54.401022 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bfff737-4fbb-4881-addc-dad872089e60 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:54.489605 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-3bfff737-4fbb-4881-addc-dad872089e60 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f705965312b541bd9e9168ea44b328b2'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:54.502816 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-3bfff737-4fbb-4881-addc-dad872089e60 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f705965312b541bd9e9168ea44b328b2'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:54.503492 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-3bfff737-4fbb-4881-addc-dad872089e60 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f705965312b541bd9e9168ea44b328b2'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:54.515711 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-3bfff737-4fbb-4881-addc-dad872089e60 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f705965312b541bd9e9168ea44b328b2'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:54.517036 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-3bfff737-4fbb-4881-addc-dad872089e60 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f705965312b541bd9e9168ea44b328b2'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:54.554851 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1154/2321] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:58:54 2022] DELETE /identity/v3/users/f705965312b541bd9e9168ea44b328b2 => generated 0 bytes in 281 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:58:54.560026 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558bb117-a82c-4b75-af7f-7a60bff620ad None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:54.561062 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558bb117-a82c-4b75-af7f-7a60bff620ad None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:54.561585 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558bb117-a82c-4b75-af7f-7a60bff620ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:54.585685 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-558bb117-a82c-4b75-af7f-7a60bff620ad None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:58:54.587195 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-558bb117-a82c-4b75-af7f-7a60bff620ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:54.678085 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1168/2322] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:58:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:58:54.697427 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bfff737-4fbb-4881-addc-dad872089e60 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:54.699035 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-3bfff737-4fbb-4881-addc-dad872089e60 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:54.778096 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bfff737-4fbb-4881-addc-dad872089e60 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:54.789727 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cdd85f5-6373-4c2b-b9a2-43f7036e1ce9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:54.790594 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cdd85f5-6373-4c2b-b9a2-43f7036e1ce9 service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:54.790946 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cdd85f5-6373-4c2b-b9a2-43f7036e1ce9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:54.791278 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cdd85f5-6373-4c2b-b9a2-43f7036e1ce9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:54.805481 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cdd85f5-6373-4c2b-b9a2-43f7036e1ce9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:54.807148 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cdd85f5-6373-4c2b-b9a2-43f7036e1ce9 service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:54.821310 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1155/2323] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:58:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:58:55.527340 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-558bb117-a82c-4b75-af7f-7a60bff620ad None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:55.528850 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-558bb117-a82c-4b75-af7f-7a60bff620ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:55.542850 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-558bb117-a82c-4b75-af7f-7a60bff620ad None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:55.547713 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04311e28-9be3-436d-8e5c-e4cf583bf17f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:55.548394 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04311e28-9be3-436d-8e5c-e4cf583bf17f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:55.548730 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04311e28-9be3-436d-8e5c-e4cf583bf17f admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:55.549002 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04311e28-9be3-436d-8e5c-e4cf583bf17f admin admin] PATH_INFO: `/v3/projects/fe51531385464725be3c70456b493e19` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:55.550248 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04311e28-9be3-436d-8e5c-e4cf583bf17f admin admin] RBAC: Authorizing `identity:delete_project(project_id=fe51531385464725be3c70456b493e19)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:55.552198 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04311e28-9be3-436d-8e5c-e4cf583bf17f admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:55.604255 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:58:55.608584 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-04311e28-9be3-436d-8e5c-e4cf583bf17f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fe51531385464725be3c70456b493e19', 'request_id': 'req-04311e28-9be3-436d-8e5c-e4cf583bf17f'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:55.618874 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-04311e28-9be3-436d-8e5c-e4cf583bf17f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fe51531385464725be3c70456b493e19', 'request_id': 'req-04311e28-9be3-436d-8e5c-e4cf583bf17f'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:55.625135 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-04311e28-9be3-436d-8e5c-e4cf583bf17f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fe51531385464725be3c70456b493e19', 'request_id': 'req-04311e28-9be3-436d-8e5c-e4cf583bf17f'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:55.637747 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-04311e28-9be3-436d-8e5c-e4cf583bf17f admin admin] The token cache is being invalidate because project fe51531385464725be3c70456b493e19 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:58:55.638112 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-04311e28-9be3-436d-8e5c-e4cf583bf17f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:55.638771 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-04311e28-9be3-436d-8e5c-e4cf583bf17f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:55.640308 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1169/2324] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:58:55 2022] DELETE /identity/v3/projects/fe51531385464725be3c70456b493e19 => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:58:55.645246 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cdd85f5-6373-4c2b-b9a2-43f7036e1ce9 service neutron] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:55.655492 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cdd85f5-6373-4c2b-b9a2-43f7036e1ce9 service neutron] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:55.675321 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:58:55.726017 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd9f91c0-258d-44b7-9ca8-7dca10e1d117 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:55.726644 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9f91c0-258d-44b7-9ca8-7dca10e1d117 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:55.726940 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9f91c0-258d-44b7-9ca8-7dca10e1d117 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:55.727185 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9f91c0-258d-44b7-9ca8-7dca10e1d117 admin admin] PATH_INFO: `/v3/users/aab4adc8b502477494b3b0d802a470eb` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:55.728317 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd9f91c0-258d-44b7-9ca8-7dca10e1d117 admin admin] RBAC: Authorizing `identity:delete_user(user_id=aab4adc8b502477494b3b0d802a470eb)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:55.730273 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd9f91c0-258d-44b7-9ca8-7dca10e1d117 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:55.778117 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-cd9f91c0-258d-44b7-9ca8-7dca10e1d117 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'aab4adc8b502477494b3b0d802a470eb'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:55.786388 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-cd9f91c0-258d-44b7-9ca8-7dca10e1d117 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'aab4adc8b502477494b3b0d802a470eb'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:55.787255 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-cd9f91c0-258d-44b7-9ca8-7dca10e1d117 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'aab4adc8b502477494b3b0d802a470eb'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:55.795104 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-cd9f91c0-258d-44b7-9ca8-7dca10e1d117 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'aab4adc8b502477494b3b0d802a470eb'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:55.795714 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-cd9f91c0-258d-44b7-9ca8-7dca10e1d117 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'aab4adc8b502477494b3b0d802a470eb'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:55.813067 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1156/2325] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:58:55 2022] DELETE /identity/v3/users/aab4adc8b502477494b3b0d802a470eb => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:58:56.090393 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04311e28-9be3-436d-8e5c-e4cf583bf17f admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:56.091833 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-04311e28-9be3-436d-8e5c-e4cf583bf17f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:56.105480 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04311e28-9be3-436d-8e5c-e4cf583bf17f admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:56.165107 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eff9c36-192d-4a06-b3a8-c301d383b66a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:56.165771 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eff9c36-192d-4a06-b3a8-c301d383b66a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:56.166050 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eff9c36-192d-4a06-b3a8-c301d383b66a admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:56.166318 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eff9c36-192d-4a06-b3a8-c301d383b66a admin admin] PATH_INFO: `/v3/projects/5883265470f0434a87df7adc31ceff1e` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:56.167453 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eff9c36-192d-4a06-b3a8-c301d383b66a admin admin] RBAC: Authorizing `identity:delete_project(project_id=5883265470f0434a87df7adc31ceff1e)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:56.169534 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eff9c36-192d-4a06-b3a8-c301d383b66a admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:56.226960 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-1eff9c36-192d-4a06-b3a8-c301d383b66a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5883265470f0434a87df7adc31ceff1e', 'request_id': 'req-1eff9c36-192d-4a06-b3a8-c301d383b66a'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:56.235610 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-1eff9c36-192d-4a06-b3a8-c301d383b66a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5883265470f0434a87df7adc31ceff1e', 'request_id': 'req-1eff9c36-192d-4a06-b3a8-c301d383b66a'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:56.244928 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-1eff9c36-192d-4a06-b3a8-c301d383b66a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5883265470f0434a87df7adc31ceff1e', 'request_id': 'req-1eff9c36-192d-4a06-b3a8-c301d383b66a'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:56.258395 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-1eff9c36-192d-4a06-b3a8-c301d383b66a admin admin] The token cache is being invalidate because project 5883265470f0434a87df7adc31ceff1e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:58:56.258949 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-1eff9c36-192d-4a06-b3a8-c301d383b66a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:56.259761 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-1eff9c36-192d-4a06-b3a8-c301d383b66a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:56.261492 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1170/2326] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:58:56 2022] DELETE /identity/v3/projects/5883265470f0434a87df7adc31ceff1e => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:58:56.267108 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5723ffb0-d1d3-47de-85af-b433354ad02b None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:56.267658 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5723ffb0-d1d3-47de-85af-b433354ad02b None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:56.267932 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5723ffb0-d1d3-47de-85af-b433354ad02b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:56.295709 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-5723ffb0-d1d3-47de-85af-b433354ad02b None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:58:56.296920 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-5723ffb0-d1d3-47de-85af-b433354ad02b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:56.382613 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1157/2327] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:58:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 00:58:56.384588 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3cd4c4f-b58b-4e48-89ba-43a86d1953f3 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:56.384869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3cd4c4f-b58b-4e48-89ba-43a86d1953f3 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:56.385144 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3cd4c4f-b58b-4e48-89ba-43a86d1953f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:56.407731 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-a3cd4c4f-b58b-4e48-89ba-43a86d1953f3 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:58:56.408901 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-a3cd4c4f-b58b-4e48-89ba-43a86d1953f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:56.418262 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1171/2328] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:58:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:58:56.422130 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5723ffb0-d1d3-47de-85af-b433354ad02b None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:56.434094 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5723ffb0-d1d3-47de-85af-b433354ad02b None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:56.440276 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5ae8b9-1356-4fa6-9a02-b92d5645b5f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:56.440888 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5ae8b9-1356-4fa6-9a02-b92d5645b5f7 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:56.441161 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5ae8b9-1356-4fa6-9a02-b92d5645b5f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:56.441456 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5ae8b9-1356-4fa6-9a02-b92d5645b5f7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:56.442112 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5ae8b9-1356-4fa6-9a02-b92d5645b5f7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:58:56.442412 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5ae8b9-1356-4fa6-9a02-b92d5645b5f7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:56.444189 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5ae8b9-1356-4fa6-9a02-b92d5645b5f7 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:56.458719 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1158/2329] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 00:58:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:58:56.462858 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3cd4c4f-b58b-4e48-89ba-43a86d1953f3 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:56.476175 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3cd4c4f-b58b-4e48-89ba-43a86d1953f3 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:56.481013 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a48fedd-3257-46c5-9156-63c5a26997ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:56.481608 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a48fedd-3257-46c5-9156-63c5a26997ca admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:56.481880 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a48fedd-3257-46c5-9156-63c5a26997ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:56.482145 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a48fedd-3257-46c5-9156-63c5a26997ca admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:56.482927 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a48fedd-3257-46c5-9156-63c5a26997ca admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:56.484705 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a48fedd-3257-46c5-9156-63c5a26997ca admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:56.520652 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1172/2330] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:58:56 2022] POST /identity/v3/projects => generated 385 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:58:56.524625 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5ae8b9-1356-4fa6-9a02-b92d5645b5f7 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:56.541377 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5ae8b9-1356-4fa6-9a02-b92d5645b5f7 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:56.570421 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:58:56.618950 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e926673f-7bf8-4036-900d-b7dbcdf6caa2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:56.619556 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e926673f-7bf8-4036-900d-b7dbcdf6caa2 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:56.619833 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e926673f-7bf8-4036-900d-b7dbcdf6caa2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:56.620101 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e926673f-7bf8-4036-900d-b7dbcdf6caa2 admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:56.620738 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e926673f-7bf8-4036-900d-b7dbcdf6caa2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:56.622709 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e926673f-7bf8-4036-900d-b7dbcdf6caa2 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:56.650315 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1159/2331] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:58:56 2022] POST /identity/v3/users => generated 293 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:58:56.654701 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a48fedd-3257-46c5-9156-63c5a26997ca admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:56.666724 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a48fedd-3257-46c5-9156-63c5a26997ca admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:56.672475 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ccf8b2f-4a02-4c76-b72f-cb8b2d403d1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:56.672703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ccf8b2f-4a02-4c76-b72f-cb8b2d403d1b admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:56.673022 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ccf8b2f-4a02-4c76-b72f-cb8b2d403d1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:56.673332 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ccf8b2f-4a02-4c76-b72f-cb8b2d403d1b admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:56.674096 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ccf8b2f-4a02-4c76-b72f-cb8b2d403d1b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:56.675766 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ccf8b2f-4a02-4c76-b72f-cb8b2d403d1b admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:56.686404 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1173/2332] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:58:56 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:58:56.691504 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e926673f-7bf8-4036-900d-b7dbcdf6caa2 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:56.701992 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e926673f-7bf8-4036-900d-b7dbcdf6caa2 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:56.708058 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11396992-62ad-44a5-b6ca-0ad4fc216615 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:56.708672 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11396992-62ad-44a5-b6ca-0ad4fc216615 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:56.708952 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11396992-62ad-44a5-b6ca-0ad4fc216615 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:56.709221 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11396992-62ad-44a5-b6ca-0ad4fc216615 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:56.709909 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11396992-62ad-44a5-b6ca-0ad4fc216615 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:56.711525 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11396992-62ad-44a5-b6ca-0ad4fc216615 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:56.723144 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1160/2333] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:58:56 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:58:56.727327 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ccf8b2f-4a02-4c76-b72f-cb8b2d403d1b admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:56.738439 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ccf8b2f-4a02-4c76-b72f-cb8b2d403d1b admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:56.743133 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9aea728-14cd-4f9a-b95b-33ca171276c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:56.743782 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9aea728-14cd-4f9a-b95b-33ca171276c6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:56.744058 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9aea728-14cd-4f9a-b95b-33ca171276c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:56.744330 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9aea728-14cd-4f9a-b95b-33ca171276c6 admin admin] PATH_INFO: `/v3/projects/47c1ee73157a462fa2f49d2cba00fbb5/users/336cc9c86d7c4694a6b648c02719d773/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:56.760117 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9aea728-14cd-4f9a-b95b-33ca171276c6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=47c1ee73157a462fa2f49d2cba00fbb5, user_id=336cc9c86d7c4694a6b648c02719d773, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:56.762881 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9aea728-14cd-4f9a-b95b-33ca171276c6 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:56.772922 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:DELETE=11,keystone:SELECT=85,keystone:INSERT=4 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:58:56.778906 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1174/2334] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:58:56 2022] PUT /identity/v3/projects/47c1ee73157a462fa2f49d2cba00fbb5/users/336cc9c86d7c4694a6b648c02719d773/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:58:56.789788 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf686ca-aea2-4229-864b-97f7ecb0ba5f None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:56.790066 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf686ca-aea2-4229-864b-97f7ecb0ba5f None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:56.790352 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf686ca-aea2-4229-864b-97f7ecb0ba5f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:56.823863 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-caf686ca-aea2-4229-864b-97f7ecb0ba5f None None] MFA Rules not processed for user `336cc9c86d7c4694a6b648c02719d773`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:58:56.825088 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-caf686ca-aea2-4229-864b-97f7ecb0ba5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:56.890115 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1161/2335] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:58:56 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:58:56.901270 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9aea728-14cd-4f9a-b95b-33ca171276c6 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:56.903015 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-d9aea728-14cd-4f9a-b95b-33ca171276c6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:56.961125 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9aea728-14cd-4f9a-b95b-33ca171276c6 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:56.964836 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b03bfc75-f6be-4399-995a-da1f8db21173 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:56.965468 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03bfc75-f6be-4399-995a-da1f8db21173 service cinder] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:56.965749 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03bfc75-f6be-4399-995a-da1f8db21173 service cinder] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:56.966016 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03bfc75-f6be-4399-995a-da1f8db21173 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:56.975339 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b03bfc75-f6be-4399-995a-da1f8db21173 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:56.976759 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b03bfc75-f6be-4399-995a-da1f8db21173 service cinder] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:56.985829 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1175/2336] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:58:56 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:58:57.005149 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf686ca-aea2-4229-864b-97f7ecb0ba5f None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:57.068119 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf686ca-aea2-4229-864b-97f7ecb0ba5f None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:57.072680 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb71fe9-38c1-4d24-ad83-ead21a62e763 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:57.073318 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb71fe9-38c1-4d24-ad83-ead21a62e763 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:57.073594 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb71fe9-38c1-4d24-ad83-ead21a62e763 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:57.073857 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb71fe9-38c1-4d24-ad83-ead21a62e763 admin admin] PATH_INFO: `/v3/users/336cc9c86d7c4694a6b648c02719d773` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:57.074973 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb71fe9-38c1-4d24-ad83-ead21a62e763 admin admin] RBAC: Authorizing `identity:delete_user(user_id=336cc9c86d7c4694a6b648c02719d773)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:57.076869 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb71fe9-38c1-4d24-ad83-ead21a62e763 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:57.121347 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=92,keystone:INSERT=3,keystone:DELETE=5 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:58:57.128468 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-fbb71fe9-38c1-4d24-ad83-ead21a62e763 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '336cc9c86d7c4694a6b648c02719d773'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:57.136490 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-fbb71fe9-38c1-4d24-ad83-ead21a62e763 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '336cc9c86d7c4694a6b648c02719d773'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:57.137112 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-fbb71fe9-38c1-4d24-ad83-ead21a62e763 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '336cc9c86d7c4694a6b648c02719d773'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:57.144689 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-fbb71fe9-38c1-4d24-ad83-ead21a62e763 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '336cc9c86d7c4694a6b648c02719d773'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:57.145476 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-fbb71fe9-38c1-4d24-ad83-ead21a62e763 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '336cc9c86d7c4694a6b648c02719d773'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:57.168767 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1162/2337] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:58:57 2022] DELETE /identity/v3/users/336cc9c86d7c4694a6b648c02719d773 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:58:57.173975 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b99b1ea-d1fa-4bdc-b2b8-a486ec5a0508 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:57.174453 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b99b1ea-d1fa-4bdc-b2b8-a486ec5a0508 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:57.174727 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b99b1ea-d1fa-4bdc-b2b8-a486ec5a0508 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:57.199232 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-5b99b1ea-d1fa-4bdc-b2b8-a486ec5a0508 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:58:57.200465 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-5b99b1ea-d1fa-4bdc-b2b8-a486ec5a0508 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:57.262667 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1176/2338] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:58:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:58:57.272788 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb71fe9-38c1-4d24-ad83-ead21a62e763 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:57.274375 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-fbb71fe9-38c1-4d24-ad83-ead21a62e763 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:57.296446 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b99b1ea-d1fa-4bdc-b2b8-a486ec5a0508 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:57.299115 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-5b99b1ea-d1fa-4bdc-b2b8-a486ec5a0508 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:57.378231 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb71fe9-38c1-4d24-ad83-ead21a62e763 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:57.382116 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee31208-20bb-469e-8b12-4ba830149af1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:57.382795 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee31208-20bb-469e-8b12-4ba830149af1 service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:57.383078 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee31208-20bb-469e-8b12-4ba830149af1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:57.383347 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee31208-20bb-469e-8b12-4ba830149af1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:57.395362 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b99b1ea-d1fa-4bdc-b2b8-a486ec5a0508 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:57.399341 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ee31208-20bb-469e-8b12-4ba830149af1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:57.400870 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ee31208-20bb-469e-8b12-4ba830149af1 service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:57.403658 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80606f72-4cba-4815-a320-9293ae0e6d7a service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '11a80b55dcd444df901c8bf80f2a2c26', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:57.404682 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80606f72-4cba-4815-a320-9293ae0e6d7a service placement] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:57.405118 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80606f72-4cba-4815-a320-9293ae0e6d7a service placement] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:57.405585 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80606f72-4cba-4815-a320-9293ae0e6d7a service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:57.408618 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-80606f72-4cba-4815-a320-9293ae0e6d7a service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:57.417323 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1163/2339] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:58:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:58:57.423440 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80606f72-4cba-4815-a320-9293ae0e6d7a service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:57.425426 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80606f72-4cba-4815-a320-9293ae0e6d7a service placement] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:57.436405 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1177/2340] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:58:57 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 00:58:57.865361 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee31208-20bb-469e-8b12-4ba830149af1 service neutron] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:57.866880 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-2ee31208-20bb-469e-8b12-4ba830149af1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:57.888359 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee31208-20bb-469e-8b12-4ba830149af1 service neutron] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:57.895764 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b663e156-b111-4247-8b8b-d6e9c1090980 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:57.896755 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b663e156-b111-4247-8b8b-d6e9c1090980 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:57.897144 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b663e156-b111-4247-8b8b-d6e9c1090980 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:57.897565 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b663e156-b111-4247-8b8b-d6e9c1090980 admin admin] PATH_INFO: `/v3/projects/47c1ee73157a462fa2f49d2cba00fbb5` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:57.899393 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b663e156-b111-4247-8b8b-d6e9c1090980 admin admin] RBAC: Authorizing `identity:delete_project(project_id=47c1ee73157a462fa2f49d2cba00fbb5)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:57.901519 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b663e156-b111-4247-8b8b-d6e9c1090980 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:57.951436 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-b663e156-b111-4247-8b8b-d6e9c1090980 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '47c1ee73157a462fa2f49d2cba00fbb5', 'request_id': 'req-b663e156-b111-4247-8b8b-d6e9c1090980'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:57.959723 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-b663e156-b111-4247-8b8b-d6e9c1090980 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '47c1ee73157a462fa2f49d2cba00fbb5', 'request_id': 'req-b663e156-b111-4247-8b8b-d6e9c1090980'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:57.966589 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-b663e156-b111-4247-8b8b-d6e9c1090980 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '47c1ee73157a462fa2f49d2cba00fbb5', 'request_id': 'req-b663e156-b111-4247-8b8b-d6e9c1090980'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:57.978571 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-b663e156-b111-4247-8b8b-d6e9c1090980 admin admin] The token cache is being invalidate because project 47c1ee73157a462fa2f49d2cba00fbb5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81386) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:58:57.978856 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-b663e156-b111-4247-8b8b-d6e9c1090980 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:57.979514 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-b663e156-b111-4247-8b8b-d6e9c1090980 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:58:57.981037 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1164/2341] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:58:57 2022] DELETE /identity/v3/projects/47c1ee73157a462fa2f49d2cba00fbb5 => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:58:57.987065 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bded5c58-20c9-4089-805e-22a7875f756e None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:57.987559 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bded5c58-20c9-4089-805e-22a7875f756e None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:57.987663 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bded5c58-20c9-4089-805e-22a7875f756e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:58.010435 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-bded5c58-20c9-4089-805e-22a7875f756e None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:58:58.011664 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-bded5c58-20c9-4089-805e-22a7875f756e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:58.044240 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:58:58.074459 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1178/2342] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:58:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:58:58.083749 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc177d5-2535-46b2-b697-99fff1a3e81a None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:58.084023 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc177d5-2535-46b2-b697-99fff1a3e81a None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:58.084287 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc177d5-2535-46b2-b697-99fff1a3e81a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:58.106505 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-5bc177d5-2535-46b2-b697-99fff1a3e81a None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:58:58.107691 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-5bc177d5-2535-46b2-b697-99fff1a3e81a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:58.118227 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1165/2343] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:58:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:58:58.122922 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bded5c58-20c9-4089-805e-22a7875f756e None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:58.132668 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bded5c58-20c9-4089-805e-22a7875f756e None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:58.137799 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89979fe-1e91-4057-a126-b2488f5bedbd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:58.138459 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89979fe-1e91-4057-a126-b2488f5bedbd admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:58.138703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89979fe-1e91-4057-a126-b2488f5bedbd admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:58.138964 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89979fe-1e91-4057-a126-b2488f5bedbd admin admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:58.139743 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d89979fe-1e91-4057-a126-b2488f5bedbd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:58:58.140044 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d89979fe-1e91-4057-a126-b2488f5bedbd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:58.141791 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d89979fe-1e91-4057-a126-b2488f5bedbd admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:58.153799 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1179/2344] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 00:58:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:58:58.158921 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bc177d5-2535-46b2-b697-99fff1a3e81a None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:58.168012 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bc177d5-2535-46b2-b697-99fff1a3e81a None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:58.173167 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a451f5a-500b-4bbe-af06-272e9a5e2867 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:58.174030 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a451f5a-500b-4bbe-af06-272e9a5e2867 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:58.174298 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a451f5a-500b-4bbe-af06-272e9a5e2867 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:58.174564 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a451f5a-500b-4bbe-af06-272e9a5e2867 admin admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:58.175196 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a451f5a-500b-4bbe-af06-272e9a5e2867 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:58.177070 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a451f5a-500b-4bbe-af06-272e9a5e2867 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:58.206960 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1166/2345] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:58:58 2022] POST /identity/v3/projects => generated 385 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:58:58.211561 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89979fe-1e91-4057-a126-b2488f5bedbd admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:58.222539 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89979fe-1e91-4057-a126-b2488f5bedbd admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:58.283723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d9e26b-9cc4-414c-9b64-3e291c08d9c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:58.284332 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d9e26b-9cc4-414c-9b64-3e291c08d9c9 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:58.284608 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d9e26b-9cc4-414c-9b64-3e291c08d9c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:58.284881 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d9e26b-9cc4-414c-9b64-3e291c08d9c9 admin admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:58.285556 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76d9e26b-9cc4-414c-9b64-3e291c08d9c9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:58.287456 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76d9e26b-9cc4-414c-9b64-3e291c08d9c9 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:58.310427 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1180/2346] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:58:58 2022] POST /identity/v3/users => generated 293 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:58:58.315472 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a451f5a-500b-4bbe-af06-272e9a5e2867 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:58.326210 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a451f5a-500b-4bbe-af06-272e9a5e2867 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:58.331115 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a772248c-268a-495c-8d9e-50ee87e6e82b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:58.331717 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a772248c-268a-495c-8d9e-50ee87e6e82b admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:58.331990 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a772248c-268a-495c-8d9e-50ee87e6e82b admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:58.332261 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a772248c-268a-495c-8d9e-50ee87e6e82b admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:58.332924 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a772248c-268a-495c-8d9e-50ee87e6e82b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:58.334548 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a772248c-268a-495c-8d9e-50ee87e6e82b admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:58.345473 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1167/2347] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:58:58 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:58:58.349734 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d9e26b-9cc4-414c-9b64-3e291c08d9c9 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:58.360703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d9e26b-9cc4-414c-9b64-3e291c08d9c9 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:58.366256 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9531d60-358b-4c80-969e-4b32a59d6543 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:58.366862 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9531d60-358b-4c80-969e-4b32a59d6543 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:58.367134 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9531d60-358b-4c80-969e-4b32a59d6543 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:58.367399 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9531d60-358b-4c80-969e-4b32a59d6543 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:58.368041 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9531d60-358b-4c80-969e-4b32a59d6543 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:58.369670 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9531d60-358b-4c80-969e-4b32a59d6543 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:58.381495 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1181/2348] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:58:58 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:58:58.385937 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a772248c-268a-495c-8d9e-50ee87e6e82b admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:58.391375 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:DELETE=12,keystone:SELECT=85,keystone:INSERT=3 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:58:58.398672 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a772248c-268a-495c-8d9e-50ee87e6e82b admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:58.405844 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c33fee1-251b-455a-b2ac-bbd4e36eeb56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:58.406523 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c33fee1-251b-455a-b2ac-bbd4e36eeb56 admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:58.407328 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c33fee1-251b-455a-b2ac-bbd4e36eeb56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:58.407644 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c33fee1-251b-455a-b2ac-bbd4e36eeb56 admin admin] PATH_INFO: `/v3/projects/8947f6ece4f4436782be77d7bd8f20dd/users/58080c4784a343438216d0191d7b4c0e/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:58.426797 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c33fee1-251b-455a-b2ac-bbd4e36eeb56 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8947f6ece4f4436782be77d7bd8f20dd, user_id=58080c4784a343438216d0191d7b4c0e, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:58.429428 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c33fee1-251b-455a-b2ac-bbd4e36eeb56 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:58.446739 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1168/2349] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:58:58 2022] PUT /identity/v3/projects/8947f6ece4f4436782be77d7bd8f20dd/users/58080c4784a343438216d0191d7b4c0e/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:58:58.458854 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8525b39-2403-4d3a-837c-6f82e19bf1d1 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:58.459042 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8525b39-2403-4d3a-837c-6f82e19bf1d1 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:58.459442 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8525b39-2403-4d3a-837c-6f82e19bf1d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:58.492877 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-a8525b39-2403-4d3a-837c-6f82e19bf1d1 None None] MFA Rules not processed for user `58080c4784a343438216d0191d7b4c0e`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:58:58.494536 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-a8525b39-2403-4d3a-837c-6f82e19bf1d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:58.563519 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1182/2350] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:58:58 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:58:58.568077 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c33fee1-251b-455a-b2ac-bbd4e36eeb56 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:58.631278 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c33fee1-251b-455a-b2ac-bbd4e36eeb56 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:58.636075 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47fddf7c-6376-474b-899b-b057f7e050be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:58.636677 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fddf7c-6376-474b-899b-b057f7e050be admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:58.636951 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fddf7c-6376-474b-899b-b057f7e050be admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:58.637218 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fddf7c-6376-474b-899b-b057f7e050be admin admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:58.637874 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47fddf7c-6376-474b-899b-b057f7e050be admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:58.639771 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47fddf7c-6376-474b-899b-b057f7e050be admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:58.669572 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1169/2351] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:58:58 2022] POST /identity/v3/projects => generated 383 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:58:58.675593 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8525b39-2403-4d3a-837c-6f82e19bf1d1 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:58.687915 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8525b39-2403-4d3a-837c-6f82e19bf1d1 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:58.720501 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:58:58.754199 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6cacaf6-dc2d-4060-80b6-e80d12cdf495 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:58.754809 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6cacaf6-dc2d-4060-80b6-e80d12cdf495 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:58.755085 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6cacaf6-dc2d-4060-80b6-e80d12cdf495 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:58.755579 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6cacaf6-dc2d-4060-80b6-e80d12cdf495 admin admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:58.756114 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6cacaf6-dc2d-4060-80b6-e80d12cdf495 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:58.758239 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6cacaf6-dc2d-4060-80b6-e80d12cdf495 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:58.780723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1183/2352] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:58:58 2022] POST /identity/v3/users => generated 292 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:58:58.786379 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47fddf7c-6376-474b-899b-b057f7e050be admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:58.799308 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47fddf7c-6376-474b-899b-b057f7e050be admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:58.804325 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f3736c-4e26-49a1-a6e1-f9103d4338bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:58.804933 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f3736c-4e26-49a1-a6e1-f9103d4338bd admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:58.805211 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f3736c-4e26-49a1-a6e1-f9103d4338bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:58.805509 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f3736c-4e26-49a1-a6e1-f9103d4338bd admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:58.806320 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33f3736c-4e26-49a1-a6e1-f9103d4338bd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:58.807827 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33f3736c-4e26-49a1-a6e1-f9103d4338bd admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:58.819418 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1170/2353] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:58:58 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:58:58.823587 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6cacaf6-dc2d-4060-80b6-e80d12cdf495 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:58.834810 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6cacaf6-dc2d-4060-80b6-e80d12cdf495 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:58.840710 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8486b73f-b03b-4bd7-a39c-6e275bc790d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:58.841365 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8486b73f-b03b-4bd7-a39c-6e275bc790d5 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:58.841643 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8486b73f-b03b-4bd7-a39c-6e275bc790d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:58.841913 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8486b73f-b03b-4bd7-a39c-6e275bc790d5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:58.842560 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8486b73f-b03b-4bd7-a39c-6e275bc790d5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:58.844650 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8486b73f-b03b-4bd7-a39c-6e275bc790d5 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:58.887519 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1184/2354] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:58:58 2022] GET /identity/v3/roles => generated 1420 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:58:58.892610 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f3736c-4e26-49a1-a6e1-f9103d4338bd admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:58.907357 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f3736c-4e26-49a1-a6e1-f9103d4338bd admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:58.913063 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bcf7e3e-be47-4041-aaf8-62a154172df2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:58.914614 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bcf7e3e-be47-4041-aaf8-62a154172df2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:58.914963 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bcf7e3e-be47-4041-aaf8-62a154172df2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:58.915462 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bcf7e3e-be47-4041-aaf8-62a154172df2 admin admin] PATH_INFO: `/v3/projects/8ecf6339cdd841b9baa6a825df16ae79/users/3b9a9e6b4e0d4129b731aa980bbf4e64/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:58.940001 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bcf7e3e-be47-4041-aaf8-62a154172df2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8ecf6339cdd841b9baa6a825df16ae79, user_id=3b9a9e6b4e0d4129b731aa980bbf4e64, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:58.943253 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bcf7e3e-be47-4041-aaf8-62a154172df2 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:58.962587 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1171/2355] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:58:58 2022] PUT /identity/v3/projects/8ecf6339cdd841b9baa6a825df16ae79/users/3b9a9e6b4e0d4129b731aa980bbf4e64/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 71 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:58:58.973984 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a9e2752-2acd-4c96-bb06-2dde1ed29273 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:58.974271 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a9e2752-2acd-4c96-bb06-2dde1ed29273 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:58.974544 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a9e2752-2acd-4c96-bb06-2dde1ed29273 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:59.009396 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-4a9e2752-2acd-4c96-bb06-2dde1ed29273 None None] MFA Rules not processed for user `3b9a9e6b4e0d4129b731aa980bbf4e64`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:58:59.010703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-4a9e2752-2acd-4c96-bb06-2dde1ed29273 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:59.096627 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1185/2356] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:58:58 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:58:59.101140 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bcf7e3e-be47-4041-aaf8-62a154172df2 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:59.178799 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bcf7e3e-be47-4041-aaf8-62a154172df2 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:59.183955 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee2ccad-8bd4-4411-b04d-a74e9c7874e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:59.184570 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee2ccad-8bd4-4411-b04d-a74e9c7874e2 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:59.184846 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee2ccad-8bd4-4411-b04d-a74e9c7874e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:59.185135 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee2ccad-8bd4-4411-b04d-a74e9c7874e2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:59.185813 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee2ccad-8bd4-4411-b04d-a74e9c7874e2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:59.187772 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee2ccad-8bd4-4411-b04d-a74e9c7874e2 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:59.218639 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1172/2357] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:58:59 2022] POST /identity/v3/projects => generated 385 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:58:59.223360 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a9e2752-2acd-4c96-bb06-2dde1ed29273 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:59.234098 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a9e2752-2acd-4c96-bb06-2dde1ed29273 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:59.299960 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c074db85-89cb-448c-bb87-56bc632f1892 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:59.300702 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c074db85-89cb-448c-bb87-56bc632f1892 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:59.300973 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c074db85-89cb-448c-bb87-56bc632f1892 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:59.301243 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c074db85-89cb-448c-bb87-56bc632f1892 admin admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:59.302024 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c074db85-89cb-448c-bb87-56bc632f1892 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:59.303842 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c074db85-89cb-448c-bb87-56bc632f1892 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:59.328429 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1186/2358] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:58:59 2022] POST /identity/v3/users => generated 299 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:58:59.332740 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee2ccad-8bd4-4411-b04d-a74e9c7874e2 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:59.343653 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee2ccad-8bd4-4411-b04d-a74e9c7874e2 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:59.348650 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-446f4ea5-393e-477a-8f1d-9ef2d1d52548 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:59.349529 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446f4ea5-393e-477a-8f1d-9ef2d1d52548 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:59.349910 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446f4ea5-393e-477a-8f1d-9ef2d1d52548 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:59.350240 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446f4ea5-393e-477a-8f1d-9ef2d1d52548 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:59.350969 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-446f4ea5-393e-477a-8f1d-9ef2d1d52548 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:59.352735 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-446f4ea5-393e-477a-8f1d-9ef2d1d52548 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:59.368859 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1173/2359] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:58:59 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 00:58:59.370014 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c074db85-89cb-448c-bb87-56bc632f1892 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:59.381099 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c074db85-89cb-448c-bb87-56bc632f1892 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:59.386346 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f34c7757-a5bb-4e2e-85fb-3e8313b2ed3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:59.387101 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34c7757-a5bb-4e2e-85fb-3e8313b2ed3e admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:59.387380 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34c7757-a5bb-4e2e-85fb-3e8313b2ed3e admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:59.387659 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34c7757-a5bb-4e2e-85fb-3e8313b2ed3e admin admin] PATH_INFO: `/v3/projects/e66a9f6a493548839c3d2a8c24a597b3/users/5cbf881b9d45408fa2e400fbe66919d4/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:59.407886 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f34c7757-a5bb-4e2e-85fb-3e8313b2ed3e admin admin] RBAC: Authorizing `identity:create_grant(project_id=e66a9f6a493548839c3d2a8c24a597b3, user_id=5cbf881b9d45408fa2e400fbe66919d4, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:59.410605 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f34c7757-a5bb-4e2e-85fb-3e8313b2ed3e admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:59.425652 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1187/2360] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:58:59 2022] PUT /identity/v3/projects/e66a9f6a493548839c3d2a8c24a597b3/users/5cbf881b9d45408fa2e400fbe66919d4/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 00:58:59.429430 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-446f4ea5-393e-477a-8f1d-9ef2d1d52548 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:59.434676 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:58:59.441449 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-446f4ea5-393e-477a-8f1d-9ef2d1d52548 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:59.528999 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d701cd-b75d-408d-b6bd-169e244d49f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:59.529647 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d701cd-b75d-408d-b6bd-169e244d49f4 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:59.530011 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d701cd-b75d-408d-b6bd-169e244d49f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:59.530283 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d701cd-b75d-408d-b6bd-169e244d49f4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:59.530929 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92d701cd-b75d-408d-b6bd-169e244d49f4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:59.533366 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92d701cd-b75d-408d-b6bd-169e244d49f4 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:59.551235 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1174/2361] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:58:59 2022] GET /identity/v3/roles => generated 1420 bytes in 118 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:58:59.553083 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f34c7757-a5bb-4e2e-85fb-3e8313b2ed3e admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:59.565662 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:58:59.572219 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f34c7757-a5bb-4e2e-85fb-3e8313b2ed3e admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:59.580512 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c268af75-0c0b-4889-96dd-b92e517e7ef6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:59.582142 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c268af75-0c0b-4889-96dd-b92e517e7ef6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:59.582641 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c268af75-0c0b-4889-96dd-b92e517e7ef6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:59.583630 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c268af75-0c0b-4889-96dd-b92e517e7ef6 admin admin] PATH_INFO: `/v3/projects/e66a9f6a493548839c3d2a8c24a597b3/users/5cbf881b9d45408fa2e400fbe66919d4/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:59.586295 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c268af75-0c0b-4889-96dd-b92e517e7ef6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e66a9f6a493548839c3d2a8c24a597b3, user_id=5cbf881b9d45408fa2e400fbe66919d4, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:59.589837 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c268af75-0c0b-4889-96dd-b92e517e7ef6 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:59.602710 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1188/2362] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:58:59 2022] PUT /identity/v3/projects/e66a9f6a493548839c3d2a8c24a597b3/users/5cbf881b9d45408fa2e400fbe66919d4/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:58:59.604337 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970d1686-e995-49ad-ad49-24ebcdf529f1 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:59.604872 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970d1686-e995-49ad-ad49-24ebcdf529f1 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:59.605214 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970d1686-e995-49ad-ad49-24ebcdf529f1 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:59.606998 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1175/2363] 10.222.0.52 () {54 vars in 921 bytes} [Sun Sep 25 00:58:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:58:59.614445 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d89c82a-061e-4a15-9776-0b6fdd92f4f7 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:59.615222 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d89c82a-061e-4a15-9776-0b6fdd92f4f7 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:59.615717 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d89c82a-061e-4a15-9776-0b6fdd92f4f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:59.616036 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58debd8f-68a1-4458-a227-93b4519886c8 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:59.616319 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58debd8f-68a1-4458-a227-93b4519886c8 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:59.616590 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58debd8f-68a1-4458-a227-93b4519886c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:59.643610 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-58debd8f-68a1-4458-a227-93b4519886c8 None None] MFA Rules not processed for user `f142988a4dd14a5f96eacea45c046f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:58:59.646612 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-58debd8f-68a1-4458-a227-93b4519886c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:59.651648 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-2d89c82a-061e-4a15-9776-0b6fdd92f4f7 None None] MFA Rules not processed for user `5cbf881b9d45408fa2e400fbe66919d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:58:59.652987 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-2d89c82a-061e-4a15-9776-0b6fdd92f4f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:59.737783 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1176/2364] 10.222.0.52 () {58 vars in 1018 bytes} [Sun Sep 25 00:58:59 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 00:58:59.747708 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1189/2365] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:58:59 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:58:59.752931 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58debd8f-68a1-4458-a227-93b4519886c8 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:59.754715 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-58debd8f-68a1-4458-a227-93b4519886c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:59.764923 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d89c82a-061e-4a15-9776-0b6fdd92f4f7 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:58:59.766844 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-2d89c82a-061e-4a15-9776-0b6fdd92f4f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:59.820942 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58debd8f-68a1-4458-a227-93b4519886c8 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:59.826688 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e381e15-52a6-477b-b624-ae3644292419 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4ebfc0abf1b842459a44ca9324252688', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:59.827672 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e381e15-52a6-477b-b624-ae3644292419 service glance] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:59.828118 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e381e15-52a6-477b-b624-ae3644292419 service glance] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:59.828554 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e381e15-52a6-477b-b624-ae3644292419 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:59.845489 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e381e15-52a6-477b-b624-ae3644292419 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:59.846930 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d89c82a-061e-4a15-9776-0b6fdd92f4f7 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:58:59.847093 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e381e15-52a6-477b-b624-ae3644292419 service glance] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:59.850979 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736b5468-f10e-4a5e-960a-d3d894fd8d0d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:58:59.851609 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736b5468-f10e-4a5e-960a-d3d894fd8d0d service cinder] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:59.851884 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736b5468-f10e-4a5e-960a-d3d894fd8d0d service cinder] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:59.852153 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736b5468-f10e-4a5e-960a-d3d894fd8d0d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:59.859941 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1177/2366] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:58:59 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:58:59.919917 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-736b5468-f10e-4a5e-960a-d3d894fd8d0d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:58:59.921580 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-736b5468-f10e-4a5e-960a-d3d894fd8d0d service cinder] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:58:59.932776 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1190/2367] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:58:59 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 00:58:59.936591 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527bee88-8bca-4919-9a07-71a2e7eb79e2 None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:59.937079 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527bee88-8bca-4919-9a07-71a2e7eb79e2 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:59.937364 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527bee88-8bca-4919-9a07-71a2e7eb79e2 None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:59.939695 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1178/2368] 10.222.0.52 () {54 vars in 921 bytes} [Sun Sep 25 00:58:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:58:59.946061 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3caafe4-9d45-4a3d-be74-f9ba5898ab00 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:58:59.946343 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3caafe4-9d45-4a3d-be74-f9ba5898ab00 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:58:59.946649 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3caafe4-9d45-4a3d-be74-f9ba5898ab00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:58:59.968596 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-c3caafe4-9d45-4a3d-be74-f9ba5898ab00 None None] MFA Rules not processed for user `20c274e96dec478b98ef51f80ea893b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:58:59.970549 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-c3caafe4-9d45-4a3d-be74-f9ba5898ab00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:58:59.996025 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:59:00.027062 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1191/2369] 10.222.0.52 () {58 vars in 1018 bytes} [Sun Sep 25 00:58:59 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:59:00.042142 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527bee88-8bca-4919-9a07-71a2e7eb79e2 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:59:00.044962 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-527bee88-8bca-4919-9a07-71a2e7eb79e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:59:00.057887 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527bee88-8bca-4919-9a07-71a2e7eb79e2 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:59:00.062837 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e76d2bc-97b2-4b8c-b244-11c7ef5c56f8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:59:00.063400 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e76d2bc-97b2-4b8c-b244-11c7ef5c56f8 service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:00.063674 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e76d2bc-97b2-4b8c-b244-11c7ef5c56f8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:00.065314 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e76d2bc-97b2-4b8c-b244-11c7ef5c56f8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:00.092081 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e76d2bc-97b2-4b8c-b244-11c7ef5c56f8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:59:00.096125 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e76d2bc-97b2-4b8c-b244-11c7ef5c56f8 service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:59:00.108654 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1179/2370] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:59:00 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 67 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:59:00.247200 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6669e78a-6907-48d0-bbde-29bc2fe11d6d None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:00.247387 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6669e78a-6907-48d0-bbde-29bc2fe11d6d None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:00.247668 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6669e78a-6907-48d0-bbde-29bc2fe11d6d None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:00.249345 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1192/2371] 10.222.0.52 () {54 vars in 921 bytes} [Sun Sep 25 00:59:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 00:59:00.261752 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ec000d-0794-40d2-8cae-9d4a59411f48 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:00.262154 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ec000d-0794-40d2-8cae-9d4a59411f48 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:00.262524 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ec000d-0794-40d2-8cae-9d4a59411f48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:00.303535 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-c8ec000d-0794-40d2-8cae-9d4a59411f48 None None] MFA Rules not processed for user `f142988a4dd14a5f96eacea45c046f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:59:00.304838 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-c8ec000d-0794-40d2-8cae-9d4a59411f48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:59:00.327200 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1180/2372] 10.222.0.52 () {58 vars in 1018 bytes} [Sun Sep 25 00:59:00 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 66 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 00:59:00.338238 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6669e78a-6907-48d0-bbde-29bc2fe11d6d None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:59:00.351805 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6669e78a-6907-48d0-bbde-29bc2fe11d6d None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:59:00.356695 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba86488-c732-42d4-821c-6b094026b480 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:59:00.357322 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba86488-c732-42d4-821c-6b094026b480 service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:00.357653 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba86488-c732-42d4-821c-6b094026b480 service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:00.358119 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba86488-c732-42d4-821c-6b094026b480 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:00.369584 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eba86488-c732-42d4-821c-6b094026b480 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:59:00.371009 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eba86488-c732-42d4-821c-6b094026b480 service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:59:00.387006 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1193/2373] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:59:00 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:59:01.536159 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ec000d-0794-40d2-8cae-9d4a59411f48 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:59:01.546690 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ec000d-0794-40d2-8cae-9d4a59411f48 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:59:01.551047 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d7b8e01-618a-4962-8b75-f38bdd8a90a9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:59:01.551667 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7b8e01-618a-4962-8b75-f38bdd8a90a9 service cinder] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:01.551945 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7b8e01-618a-4962-8b75-f38bdd8a90a9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:01.552368 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7b8e01-618a-4962-8b75-f38bdd8a90a9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:01.562718 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:59:01.618019 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d7b8e01-618a-4962-8b75-f38bdd8a90a9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:59:01.619665 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d7b8e01-618a-4962-8b75-f38bdd8a90a9 service cinder] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:59:01.630987 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1181/2374] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:59:01 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:59:02.315701 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba86488-c732-42d4-821c-6b094026b480 service neutron] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:59:02.324974 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba86488-c732-42d4-821c-6b094026b480 service neutron] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:59:02.330807 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd9d0373-d8ef-4b49-850a-af40ab016bb1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:59:02.330807 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9d0373-d8ef-4b49-850a-af40ab016bb1 service cinder] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:02.330807 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9d0373-d8ef-4b49-850a-af40ab016bb1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:02.330807 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9d0373-d8ef-4b49-850a-af40ab016bb1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:02.340182 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd9d0373-d8ef-4b49-850a-af40ab016bb1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:59:02.341641 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd9d0373-d8ef-4b49-850a-af40ab016bb1 service cinder] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:59:02.351885 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1194/2375] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:59:02 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:59:04.496248 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b1120c-6d76-432e-b5b4-af0347987c61 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:04.496827 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b1120c-6d76-432e-b5b4-af0347987c61 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:04.496935 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b1120c-6d76-432e-b5b4-af0347987c61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:04.519692 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-96b1120c-6d76-432e-b5b4-af0347987c61 None None] MFA Rules not processed for user `3f20ffd8df994c1db1b1e7248db5b5a3`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:59:04.520954 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-96b1120c-6d76-432e-b5b4-af0347987c61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:59:04.580474 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1182/2376] 10.222.0.52 () {58 vars in 1009 bytes} [Sun Sep 25 00:59:04 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:59:04.592642 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd9d0373-d8ef-4b49-850a-af40ab016bb1 service cinder] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:59:04.594204 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-fd9d0373-d8ef-4b49-850a-af40ab016bb1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:59:04.665881 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd9d0373-d8ef-4b49-850a-af40ab016bb1 service cinder] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:59:04.670955 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b3f184-afee-4ac4-bd9e-c54075d3dc02 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da606b3906dd4171bd76fd281e03e258', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:59:04.671563 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b3f184-afee-4ac4-bd9e-c54075d3dc02 service swift] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:04.671843 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b3f184-afee-4ac4-bd9e-c54075d3dc02 service swift] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:04.672116 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b3f184-afee-4ac4-bd9e-c54075d3dc02 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:04.680509 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5b3f184-afee-4ac4-bd9e-c54075d3dc02 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:59:04.682138 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5b3f184-afee-4ac4-bd9e-c54075d3dc02 service swift] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:59:04.690256 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1195/2377] 10.222.0.52 () {60 vars in 1384 bytes} [Sun Sep 25 00:59:04 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 25 00:59:06.470846 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b1120c-6d76-432e-b5b4-af0347987c61 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:59:06.568520 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b1120c-6d76-432e-b5b4-af0347987c61 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:59:06.573945 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e003992-eecc-4b71-8e34-f8a5d0dc1528 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:59:06.574523 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e003992-eecc-4b71-8e34-f8a5d0dc1528 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:06.574776 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e003992-eecc-4b71-8e34-f8a5d0dc1528 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:06.575039 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e003992-eecc-4b71-8e34-f8a5d0dc1528 admin admin] PATH_INFO: `/v3/users/58080c4784a343438216d0191d7b4c0e` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:06.580281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e003992-eecc-4b71-8e34-f8a5d0dc1528 admin admin] RBAC: Authorizing `identity:delete_user(user_id=58080c4784a343438216d0191d7b4c0e)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:59:06.583239 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e003992-eecc-4b71-8e34-f8a5d0dc1528 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:59:06.651238 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-2e003992-eecc-4b71-8e34-f8a5d0dc1528 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '58080c4784a343438216d0191d7b4c0e'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:06.661114 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-2e003992-eecc-4b71-8e34-f8a5d0dc1528 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '58080c4784a343438216d0191d7b4c0e'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:06.664734 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-2e003992-eecc-4b71-8e34-f8a5d0dc1528 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '58080c4784a343438216d0191d7b4c0e'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:06.670227 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:59:06.676759 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-2e003992-eecc-4b71-8e34-f8a5d0dc1528 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '58080c4784a343438216d0191d7b4c0e'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:06.678018 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-2e003992-eecc-4b71-8e34-f8a5d0dc1528 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '58080c4784a343438216d0191d7b4c0e'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:06.702218 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1183/2378] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:59:06 2022] DELETE /identity/v3/users/58080c4784a343438216d0191d7b4c0e => generated 0 bytes in 233 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:59:06.707341 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f350f3-a2ef-4bcf-89a2-a607abf8be90 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:06.707943 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f350f3-a2ef-4bcf-89a2-a607abf8be90 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:06.707943 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f350f3-a2ef-4bcf-89a2-a607abf8be90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:06.728978 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-f0f350f3-a2ef-4bcf-89a2-a607abf8be90 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:59:06.730216 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-f0f350f3-a2ef-4bcf-89a2-a607abf8be90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:59:06.807199 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1196/2379] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:59:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:59:06.818671 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e003992-eecc-4b71-8e34-f8a5d0dc1528 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:59:06.819237 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-2e003992-eecc-4b71-8e34-f8a5d0dc1528 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:59:06.909805 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e003992-eecc-4b71-8e34-f8a5d0dc1528 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:59:06.935276 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec160f8a-d087-4793-9f4c-afdd38f5636b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:59:06.936568 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec160f8a-d087-4793-9f4c-afdd38f5636b service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:06.936938 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec160f8a-d087-4793-9f4c-afdd38f5636b service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:06.941051 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec160f8a-d087-4793-9f4c-afdd38f5636b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:06.957032 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec160f8a-d087-4793-9f4c-afdd38f5636b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:59:06.958943 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec160f8a-d087-4793-9f4c-afdd38f5636b service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:59:06.973003 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1184/2380] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:59:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:59:07.592783 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f350f3-a2ef-4bcf-89a2-a607abf8be90 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:59:07.594606 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-f0f350f3-a2ef-4bcf-89a2-a607abf8be90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:59:07.609701 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f350f3-a2ef-4bcf-89a2-a607abf8be90 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:59:07.615371 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61378f12-317e-450d-98f6-30f79c40ade3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:59:07.616236 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61378f12-317e-450d-98f6-30f79c40ade3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:07.616517 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61378f12-317e-450d-98f6-30f79c40ade3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:07.616783 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61378f12-317e-450d-98f6-30f79c40ade3 admin admin] PATH_INFO: `/v3/projects/8947f6ece4f4436782be77d7bd8f20dd` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:07.618234 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61378f12-317e-450d-98f6-30f79c40ade3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8947f6ece4f4436782be77d7bd8f20dd)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:59:07.620184 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61378f12-317e-450d-98f6-30f79c40ade3 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:59:07.672081 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:59:07.682978 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-61378f12-317e-450d-98f6-30f79c40ade3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8947f6ece4f4436782be77d7bd8f20dd', 'request_id': 'req-61378f12-317e-450d-98f6-30f79c40ade3'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:07.698851 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-61378f12-317e-450d-98f6-30f79c40ade3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8947f6ece4f4436782be77d7bd8f20dd', 'request_id': 'req-61378f12-317e-450d-98f6-30f79c40ade3'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:07.710723 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-61378f12-317e-450d-98f6-30f79c40ade3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8947f6ece4f4436782be77d7bd8f20dd', 'request_id': 'req-61378f12-317e-450d-98f6-30f79c40ade3'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:07.727389 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-61378f12-317e-450d-98f6-30f79c40ade3 admin admin] The token cache is being invalidate because project 8947f6ece4f4436782be77d7bd8f20dd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:59:07.727675 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-61378f12-317e-450d-98f6-30f79c40ade3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:07.728495 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-61378f12-317e-450d-98f6-30f79c40ade3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:07.730066 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1197/2381] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:59:07 2022] DELETE /identity/v3/projects/8947f6ece4f4436782be77d7bd8f20dd => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:59:07.734318 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec160f8a-d087-4793-9f4c-afdd38f5636b service neutron] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:59:07.745188 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec160f8a-d087-4793-9f4c-afdd38f5636b service neutron] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:59:07.816229 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c3112d5-8789-4993-8644-5d27c4eae916 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:59:07.817036 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3112d5-8789-4993-8644-5d27c4eae916 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:07.817438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3112d5-8789-4993-8644-5d27c4eae916 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:07.817796 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3112d5-8789-4993-8644-5d27c4eae916 admin admin] PATH_INFO: `/v3/users/3b9a9e6b4e0d4129b731aa980bbf4e64` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:07.819196 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c3112d5-8789-4993-8644-5d27c4eae916 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3b9a9e6b4e0d4129b731aa980bbf4e64)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:59:07.822267 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c3112d5-8789-4993-8644-5d27c4eae916 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:59:07.886931 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-8c3112d5-8789-4993-8644-5d27c4eae916 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3b9a9e6b4e0d4129b731aa980bbf4e64'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:07.905301 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-8c3112d5-8789-4993-8644-5d27c4eae916 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3b9a9e6b4e0d4129b731aa980bbf4e64'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:07.905912 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-8c3112d5-8789-4993-8644-5d27c4eae916 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3b9a9e6b4e0d4129b731aa980bbf4e64'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:07.918147 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-8c3112d5-8789-4993-8644-5d27c4eae916 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3b9a9e6b4e0d4129b731aa980bbf4e64'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:07.918898 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-8c3112d5-8789-4993-8644-5d27c4eae916 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3b9a9e6b4e0d4129b731aa980bbf4e64'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:07.935834 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:59:07.942429 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1185/2382] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:59:07 2022] DELETE /identity/v3/users/3b9a9e6b4e0d4129b731aa980bbf4e64 => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 00:59:08.382041 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61378f12-317e-450d-98f6-30f79c40ade3 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:59:08.383307 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-61378f12-317e-450d-98f6-30f79c40ade3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:59:08.411555 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61378f12-317e-450d-98f6-30f79c40ade3 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:59:08.490057 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea802ba-9f06-43b6-89a8-c8aec61cc2f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:59:08.490787 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea802ba-9f06-43b6-89a8-c8aec61cc2f6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:08.491129 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea802ba-9f06-43b6-89a8-c8aec61cc2f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:08.491466 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea802ba-9f06-43b6-89a8-c8aec61cc2f6 admin admin] PATH_INFO: `/v3/projects/8ecf6339cdd841b9baa6a825df16ae79` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:08.492729 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea802ba-9f06-43b6-89a8-c8aec61cc2f6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8ecf6339cdd841b9baa6a825df16ae79)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:59:08.496030 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea802ba-9f06-43b6-89a8-c8aec61cc2f6 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:59:08.551311 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-eea802ba-9f06-43b6-89a8-c8aec61cc2f6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8ecf6339cdd841b9baa6a825df16ae79', 'request_id': 'req-eea802ba-9f06-43b6-89a8-c8aec61cc2f6'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:08.561836 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-eea802ba-9f06-43b6-89a8-c8aec61cc2f6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8ecf6339cdd841b9baa6a825df16ae79', 'request_id': 'req-eea802ba-9f06-43b6-89a8-c8aec61cc2f6'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:08.568101 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-eea802ba-9f06-43b6-89a8-c8aec61cc2f6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8ecf6339cdd841b9baa6a825df16ae79', 'request_id': 'req-eea802ba-9f06-43b6-89a8-c8aec61cc2f6'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:08.579389 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-eea802ba-9f06-43b6-89a8-c8aec61cc2f6 admin admin] The token cache is being invalidate because project 8ecf6339cdd841b9baa6a825df16ae79 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:59:08.579669 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-eea802ba-9f06-43b6-89a8-c8aec61cc2f6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:08.580310 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-eea802ba-9f06-43b6-89a8-c8aec61cc2f6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:08.581814 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1198/2383] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:59:08 2022] DELETE /identity/v3/projects/8ecf6339cdd841b9baa6a825df16ae79 => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:59:08.585963 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c3112d5-8789-4993-8644-5d27c4eae916 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:59:08.587385 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-8c3112d5-8789-4993-8644-5d27c4eae916 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:59:08.601756 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c3112d5-8789-4993-8644-5d27c4eae916 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:59:08.671115 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e5f2a6-9b81-4c77-b12e-f80a0c278f84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:59:08.671815 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e5f2a6-9b81-4c77-b12e-f80a0c278f84 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:08.672643 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e5f2a6-9b81-4c77-b12e-f80a0c278f84 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:08.673021 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e5f2a6-9b81-4c77-b12e-f80a0c278f84 admin admin] PATH_INFO: `/v3/users/5cbf881b9d45408fa2e400fbe66919d4` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:08.677067 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6e5f2a6-9b81-4c77-b12e-f80a0c278f84 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5cbf881b9d45408fa2e400fbe66919d4)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:59:08.678670 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6e5f2a6-9b81-4c77-b12e-f80a0c278f84 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:59:08.738789 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-f6e5f2a6-9b81-4c77-b12e-f80a0c278f84 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5cbf881b9d45408fa2e400fbe66919d4'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:08.747215 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-f6e5f2a6-9b81-4c77-b12e-f80a0c278f84 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5cbf881b9d45408fa2e400fbe66919d4'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:08.748074 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-f6e5f2a6-9b81-4c77-b12e-f80a0c278f84 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5cbf881b9d45408fa2e400fbe66919d4'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:08.755946 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-f6e5f2a6-9b81-4c77-b12e-f80a0c278f84 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5cbf881b9d45408fa2e400fbe66919d4'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:08.756682 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-f6e5f2a6-9b81-4c77-b12e-f80a0c278f84 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5cbf881b9d45408fa2e400fbe66919d4'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:08.777041 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1186/2384] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:59:08 2022] DELETE /identity/v3/users/5cbf881b9d45408fa2e400fbe66919d4 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:59:09.088230 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea802ba-9f06-43b6-89a8-c8aec61cc2f6 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:59:09.090332 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-eea802ba-9f06-43b6-89a8-c8aec61cc2f6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:59:09.109165 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea802ba-9f06-43b6-89a8-c8aec61cc2f6 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:59:09.182249 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=87,keystone:DELETE=11,keystone:INSERT=2 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:59:09.194489 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8ee65c8-65e8-4e4d-9750-3fec8fdcf1e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:59:09.195486 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ee65c8-65e8-4e4d-9750-3fec8fdcf1e8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:09.195914 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ee65c8-65e8-4e4d-9750-3fec8fdcf1e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:09.196351 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ee65c8-65e8-4e4d-9750-3fec8fdcf1e8 admin admin] PATH_INFO: `/v3/projects/e66a9f6a493548839c3d2a8c24a597b3` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:09.197560 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8ee65c8-65e8-4e4d-9750-3fec8fdcf1e8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e66a9f6a493548839c3d2a8c24a597b3)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:59:09.199476 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8ee65c8-65e8-4e4d-9750-3fec8fdcf1e8 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:59:09.273199 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-b8ee65c8-65e8-4e4d-9750-3fec8fdcf1e8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e66a9f6a493548839c3d2a8c24a597b3', 'request_id': 'req-b8ee65c8-65e8-4e4d-9750-3fec8fdcf1e8'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:09.286904 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-b8ee65c8-65e8-4e4d-9750-3fec8fdcf1e8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e66a9f6a493548839c3d2a8c24a597b3', 'request_id': 'req-b8ee65c8-65e8-4e4d-9750-3fec8fdcf1e8'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:09.293834 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-b8ee65c8-65e8-4e4d-9750-3fec8fdcf1e8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e66a9f6a493548839c3d2a8c24a597b3', 'request_id': 'req-b8ee65c8-65e8-4e4d-9750-3fec8fdcf1e8'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:09.308370 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-b8ee65c8-65e8-4e4d-9750-3fec8fdcf1e8 admin admin] The token cache is being invalidate because project e66a9f6a493548839c3d2a8c24a597b3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:59:09.308658 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-b8ee65c8-65e8-4e4d-9750-3fec8fdcf1e8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:09.309360 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-b8ee65c8-65e8-4e4d-9750-3fec8fdcf1e8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:09.311115 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1199/2385] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:59:09 2022] DELETE /identity/v3/projects/e66a9f6a493548839c3d2a8c24a597b3 => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:59:09.316688 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf3a3660-e1d9-4fe2-b2f8-5b846929cdf8 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:09.317217 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf3a3660-e1d9-4fe2-b2f8-5b846929cdf8 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:09.317663 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf3a3660-e1d9-4fe2-b2f8-5b846929cdf8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:09.360283 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-bf3a3660-e1d9-4fe2-b2f8-5b846929cdf8 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:59:09.362272 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-bf3a3660-e1d9-4fe2-b2f8-5b846929cdf8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:59:09.470596 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1187/2386] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:59:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:59:09.479445 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ca6bd8-053e-47a7-9c5f-9b6c460ca4b1 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:09.480208 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ca6bd8-053e-47a7-9c5f-9b6c460ca4b1 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:09.480676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ca6bd8-053e-47a7-9c5f-9b6c460ca4b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:09.526006 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-d3ca6bd8-053e-47a7-9c5f-9b6c460ca4b1 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:59:09.527169 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-d3ca6bd8-053e-47a7-9c5f-9b6c460ca4b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:59:09.542148 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1200/2387] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:59:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 64 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:59:09.543258 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf3a3660-e1d9-4fe2-b2f8-5b846929cdf8 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:59:09.554920 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf3a3660-e1d9-4fe2-b2f8-5b846929cdf8 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:59:09.560113 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6947a491-82f6-4c40-9bb1-f23c04763729 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:59:09.560739 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6947a491-82f6-4c40-9bb1-f23c04763729 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:09.561017 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6947a491-82f6-4c40-9bb1-f23c04763729 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:09.561310 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6947a491-82f6-4c40-9bb1-f23c04763729 admin admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:09.562162 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6947a491-82f6-4c40-9bb1-f23c04763729 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:59:09.562466 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6947a491-82f6-4c40-9bb1-f23c04763729 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:59:09.571534 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6947a491-82f6-4c40-9bb1-f23c04763729 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:59:09.586009 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1188/2388] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 00:59:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 00:59:09.592609 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3ca6bd8-053e-47a7-9c5f-9b6c460ca4b1 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:59:09.601164 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3ca6bd8-053e-47a7-9c5f-9b6c460ca4b1 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:59:09.605883 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc2520cf-c48f-487c-9125-2d466191fad6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:59:09.606487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc2520cf-c48f-487c-9125-2d466191fad6 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:09.606760 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc2520cf-c48f-487c-9125-2d466191fad6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:09.607026 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc2520cf-c48f-487c-9125-2d466191fad6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:09.607687 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc2520cf-c48f-487c-9125-2d466191fad6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:59:09.610160 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc2520cf-c48f-487c-9125-2d466191fad6 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:59:09.643659 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1201/2389] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:59:09 2022] POST /identity/v3/projects => generated 385 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:59:09.644884 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6947a491-82f6-4c40-9bb1-f23c04763729 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:59:09.652698 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:59:09.656403 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6947a491-82f6-4c40-9bb1-f23c04763729 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:59:09.724925 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a326a397-9ae9-4834-a840-7761b7148585 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:59:09.725610 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a326a397-9ae9-4834-a840-7761b7148585 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:09.729626 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a326a397-9ae9-4834-a840-7761b7148585 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:09.730145 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a326a397-9ae9-4834-a840-7761b7148585 admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:09.731327 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a326a397-9ae9-4834-a840-7761b7148585 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:59:09.736161 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a326a397-9ae9-4834-a840-7761b7148585 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:59:09.761826 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1189/2390] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:59:09 2022] POST /identity/v3/users => generated 293 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:59:09.767351 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc2520cf-c48f-487c-9125-2d466191fad6 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:59:09.778358 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc2520cf-c48f-487c-9125-2d466191fad6 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:59:09.784727 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d09953ca-02d3-4262-8c18-8be172deadec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:59:09.785473 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09953ca-02d3-4262-8c18-8be172deadec admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:09.785810 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09953ca-02d3-4262-8c18-8be172deadec admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:09.786139 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09953ca-02d3-4262-8c18-8be172deadec admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:09.786910 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d09953ca-02d3-4262-8c18-8be172deadec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:59:09.788688 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d09953ca-02d3-4262-8c18-8be172deadec admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:59:09.800476 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1202/2391] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:59:09 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:59:09.804812 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a326a397-9ae9-4834-a840-7761b7148585 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:59:09.818191 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a326a397-9ae9-4834-a840-7761b7148585 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:59:09.822966 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf2a6d9-df37-416c-b40b-d3619c40b594 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:59:09.824112 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf2a6d9-df37-416c-b40b-d3619c40b594 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:09.824112 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf2a6d9-df37-416c-b40b-d3619c40b594 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:09.824317 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf2a6d9-df37-416c-b40b-d3619c40b594 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:09.824985 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cf2a6d9-df37-416c-b40b-d3619c40b594 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:59:09.826691 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cf2a6d9-df37-416c-b40b-d3619c40b594 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:59:09.839376 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1190/2392] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:59:09 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 00:59:09.843859 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d09953ca-02d3-4262-8c18-8be172deadec admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:59:09.854819 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d09953ca-02d3-4262-8c18-8be172deadec admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:59:09.859652 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1537811f-5220-4343-8dbf-b930f84f55e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:59:09.860556 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1537811f-5220-4343-8dbf-b930f84f55e9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:09.860895 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1537811f-5220-4343-8dbf-b930f84f55e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:09.861233 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1537811f-5220-4343-8dbf-b930f84f55e9 admin admin] PATH_INFO: `/v3/projects/9c8f88cd571d4d64a04c4ba80c5f3530/users/749802ea869746f78d56843098dc7e0a/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:09.878631 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1537811f-5220-4343-8dbf-b930f84f55e9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9c8f88cd571d4d64a04c4ba80c5f3530, user_id=749802ea869746f78d56843098dc7e0a, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:59:09.881288 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1537811f-5220-4343-8dbf-b930f84f55e9 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:59:09.896578 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1203/2393] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:59:09 2022] PUT /identity/v3/projects/9c8f88cd571d4d64a04c4ba80c5f3530/users/749802ea869746f78d56843098dc7e0a/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:59:09.908221 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a2f598-4f3d-4a65-8b4d-14d784029ca4 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:09.908501 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a2f598-4f3d-4a65-8b4d-14d784029ca4 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:09.908769 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a2f598-4f3d-4a65-8b4d-14d784029ca4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:09.942228 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-89a2f598-4f3d-4a65-8b4d-14d784029ca4 None None] MFA Rules not processed for user `749802ea869746f78d56843098dc7e0a`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:59:09.942228 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-89a2f598-4f3d-4a65-8b4d-14d784029ca4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:59:10.021465 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1191/2394] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:59:09 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:59:10.030201 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1537811f-5220-4343-8dbf-b930f84f55e9 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:59:10.032525 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-1537811f-5220-4343-8dbf-b930f84f55e9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:59:10.092713 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1537811f-5220-4343-8dbf-b930f84f55e9 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:59:10.096999 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3cf21c-ce46-40ee-a88f-9acc144922be service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:59:10.097639 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3cf21c-ce46-40ee-a88f-9acc144922be service cinder] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:10.097918 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3cf21c-ce46-40ee-a88f-9acc144922be service cinder] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:10.098190 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3cf21c-ce46-40ee-a88f-9acc144922be service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:10.107719 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e3cf21c-ce46-40ee-a88f-9acc144922be service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:59:10.109296 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e3cf21c-ce46-40ee-a88f-9acc144922be service cinder] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:59:10.120288 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1204/2395] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:59:10 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:59:12.845060 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89a2f598-4f3d-4a65-8b4d-14d784029ca4 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:59:12.917298 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89a2f598-4f3d-4a65-8b4d-14d784029ca4 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:59:12.921981 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19db1bae-f5b7-40af-a030-004eeda900ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:59:12.922627 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19db1bae-f5b7-40af-a030-004eeda900ee admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:12.922965 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19db1bae-f5b7-40af-a030-004eeda900ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:12.923233 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19db1bae-f5b7-40af-a030-004eeda900ee admin admin] PATH_INFO: `/v3/users/749802ea869746f78d56843098dc7e0a` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:12.924388 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19db1bae-f5b7-40af-a030-004eeda900ee admin admin] RBAC: Authorizing `identity:delete_user(user_id=749802ea869746f78d56843098dc7e0a)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:59:12.926366 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19db1bae-f5b7-40af-a030-004eeda900ee admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:59:12.941671 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=96,keystone:INSERT=3,keystone:DELETE=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:59:12.978198 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-19db1bae-f5b7-40af-a030-004eeda900ee admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '749802ea869746f78d56843098dc7e0a'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:12.986063 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-19db1bae-f5b7-40af-a030-004eeda900ee admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '749802ea869746f78d56843098dc7e0a'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:12.986985 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-19db1bae-f5b7-40af-a030-004eeda900ee admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '749802ea869746f78d56843098dc7e0a'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:12.997341 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-19db1bae-f5b7-40af-a030-004eeda900ee admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '749802ea869746f78d56843098dc7e0a'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:12.997652 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-19db1bae-f5b7-40af-a030-004eeda900ee admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '749802ea869746f78d56843098dc7e0a'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:13.021239 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1192/2396] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:59:12 2022] DELETE /identity/v3/users/749802ea869746f78d56843098dc7e0a => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 00:59:13.022836 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-551a7fc5-ae36-4fae-a1ab-69dba0fcd6af None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:13.023297 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-551a7fc5-ae36-4fae-a1ab-69dba0fcd6af None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:13.023388 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-551a7fc5-ae36-4fae-a1ab-69dba0fcd6af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:13.044469 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-551a7fc5-ae36-4fae-a1ab-69dba0fcd6af None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:59:13.047626 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-551a7fc5-ae36-4fae-a1ab-69dba0fcd6af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:59:13.096175 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:59:13.163346 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1205/2397] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:59:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:59:13.170144 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19db1bae-f5b7-40af-a030-004eeda900ee admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:59:13.176440 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-19db1bae-f5b7-40af-a030-004eeda900ee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:59:13.261416 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19db1bae-f5b7-40af-a030-004eeda900ee admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:59:13.265365 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b94dd8-e443-4f08-8784-4b8f7378be39 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:59:13.266741 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b94dd8-e443-4f08-8784-4b8f7378be39 service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:13.267018 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b94dd8-e443-4f08-8784-4b8f7378be39 service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:13.267289 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b94dd8-e443-4f08-8784-4b8f7378be39 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:13.278159 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b94dd8-e443-4f08-8784-4b8f7378be39 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:59:13.279794 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b94dd8-e443-4f08-8784-4b8f7378be39 service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:59:13.291108 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1193/2398] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:59:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:59:13.591895 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-551a7fc5-ae36-4fae-a1ab-69dba0fcd6af None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:59:13.593689 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-551a7fc5-ae36-4fae-a1ab-69dba0fcd6af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:59:13.612068 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-551a7fc5-ae36-4fae-a1ab-69dba0fcd6af None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:59:13.617100 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ef3788-4ace-48de-8e9e-46c8d046d2ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:59:13.617700 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ef3788-4ace-48de-8e9e-46c8d046d2ea admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:13.617972 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ef3788-4ace-48de-8e9e-46c8d046d2ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:13.618281 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ef3788-4ace-48de-8e9e-46c8d046d2ea admin admin] PATH_INFO: `/v3/projects/9c8f88cd571d4d64a04c4ba80c5f3530` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:13.619634 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8ef3788-4ace-48de-8e9e-46c8d046d2ea admin admin] RBAC: Authorizing `identity:delete_project(project_id=9c8f88cd571d4d64a04c4ba80c5f3530)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:59:13.621705 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8ef3788-4ace-48de-8e9e-46c8d046d2ea admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:59:13.674006 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-c8ef3788-4ace-48de-8e9e-46c8d046d2ea admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9c8f88cd571d4d64a04c4ba80c5f3530', 'request_id': 'req-c8ef3788-4ace-48de-8e9e-46c8d046d2ea'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:13.682168 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-c8ef3788-4ace-48de-8e9e-46c8d046d2ea admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9c8f88cd571d4d64a04c4ba80c5f3530', 'request_id': 'req-c8ef3788-4ace-48de-8e9e-46c8d046d2ea'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:13.689179 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-c8ef3788-4ace-48de-8e9e-46c8d046d2ea admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9c8f88cd571d4d64a04c4ba80c5f3530', 'request_id': 'req-c8ef3788-4ace-48de-8e9e-46c8d046d2ea'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:13.706237 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-c8ef3788-4ace-48de-8e9e-46c8d046d2ea admin admin] The token cache is being invalidate because project 9c8f88cd571d4d64a04c4ba80c5f3530 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:59:13.706759 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-c8ef3788-4ace-48de-8e9e-46c8d046d2ea admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:13.707815 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-c8ef3788-4ace-48de-8e9e-46c8d046d2ea admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:13.710445 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1206/2399] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:59:13 2022] DELETE /identity/v3/projects/9c8f88cd571d4d64a04c4ba80c5f3530 => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:59:13.715278 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6341fbe-7dff-45a0-91ea-3b78f655e1d7 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:13.715584 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6341fbe-7dff-45a0-91ea-3b78f655e1d7 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:13.715853 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6341fbe-7dff-45a0-91ea-3b78f655e1d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:13.739435 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-b6341fbe-7dff-45a0-91ea-3b78f655e1d7 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:59:13.740646 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-b6341fbe-7dff-45a0-91ea-3b78f655e1d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:59:13.819077 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1194/2400] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:59:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:59:13.826529 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92eb9fd-919a-4414-8b86-2d11bdb42374 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:13.826804 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92eb9fd-919a-4414-8b86-2d11bdb42374 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:13.827073 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92eb9fd-919a-4414-8b86-2d11bdb42374 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:13.850046 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-d92eb9fd-919a-4414-8b86-2d11bdb42374 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:59:13.851247 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-d92eb9fd-919a-4414-8b86-2d11bdb42374 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:59:13.859200 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1207/2401] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:59:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:59:13.863974 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6341fbe-7dff-45a0-91ea-3b78f655e1d7 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:59:13.873839 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6341fbe-7dff-45a0-91ea-3b78f655e1d7 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:59:13.879668 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cfdea94-0fa0-4901-bbcf-d01eb6250260 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:59:13.880313 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cfdea94-0fa0-4901-bbcf-d01eb6250260 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:13.880567 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cfdea94-0fa0-4901-bbcf-d01eb6250260 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:13.880830 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cfdea94-0fa0-4901-bbcf-d01eb6250260 admin admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:13.881544 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cfdea94-0fa0-4901-bbcf-d01eb6250260 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:59:13.881817 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cfdea94-0fa0-4901-bbcf-d01eb6250260 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:59:13.883443 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cfdea94-0fa0-4901-bbcf-d01eb6250260 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:59:13.895910 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1195/2402] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 00:59:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:59:13.900991 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92eb9fd-919a-4414-8b86-2d11bdb42374 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:59:13.912389 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92eb9fd-919a-4414-8b86-2d11bdb42374 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:59:13.917114 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e7db2f2-ff99-4178-b27b-39d266d3ab25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:59:13.917749 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7db2f2-ff99-4178-b27b-39d266d3ab25 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:13.918022 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7db2f2-ff99-4178-b27b-39d266d3ab25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:13.918363 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7db2f2-ff99-4178-b27b-39d266d3ab25 admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:13.919068 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e7db2f2-ff99-4178-b27b-39d266d3ab25 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:59:13.920940 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e7db2f2-ff99-4178-b27b-39d266d3ab25 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:59:13.952873 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1208/2403] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:59:13 2022] POST /identity/v3/projects => generated 375 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 00:59:13.957281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cfdea94-0fa0-4901-bbcf-d01eb6250260 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:59:13.967462 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cfdea94-0fa0-4901-bbcf-d01eb6250260 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:59:13.987811 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:59:14.030965 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d39065-ccde-4726-9d8c-c497a4e7a3ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:59:14.031616 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d39065-ccde-4726-9d8c-c497a4e7a3ca admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:14.031969 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d39065-ccde-4726-9d8c-c497a4e7a3ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:14.032236 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d39065-ccde-4726-9d8c-c497a4e7a3ca admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:14.032881 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67d39065-ccde-4726-9d8c-c497a4e7a3ca admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:59:14.034821 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67d39065-ccde-4726-9d8c-c497a4e7a3ca admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:59:14.057091 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1196/2404] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:59:13 2022] POST /identity/v3/users => generated 288 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:59:14.061025 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e7db2f2-ff99-4178-b27b-39d266d3ab25 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:59:14.072038 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e7db2f2-ff99-4178-b27b-39d266d3ab25 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:59:14.076798 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54eb1f9-7f2e-441e-a735-fea60e851175 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:59:14.077426 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54eb1f9-7f2e-441e-a735-fea60e851175 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:14.077705 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54eb1f9-7f2e-441e-a735-fea60e851175 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:14.078026 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54eb1f9-7f2e-441e-a735-fea60e851175 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:14.078809 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c54eb1f9-7f2e-441e-a735-fea60e851175 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:59:14.080395 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c54eb1f9-7f2e-441e-a735-fea60e851175 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:59:14.090661 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1209/2405] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:59:14 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:59:14.094619 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d39065-ccde-4726-9d8c-c497a4e7a3ca admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:59:14.104530 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d39065-ccde-4726-9d8c-c497a4e7a3ca admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:59:14.109127 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ccd034a-71ee-495d-828d-747620747343 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:59:14.110340 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ccd034a-71ee-495d-828d-747620747343 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:14.110609 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ccd034a-71ee-495d-828d-747620747343 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:14.110962 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ccd034a-71ee-495d-828d-747620747343 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:14.111950 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ccd034a-71ee-495d-828d-747620747343 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:59:14.113525 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ccd034a-71ee-495d-828d-747620747343 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:59:14.124370 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1197/2406] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:59:14 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 00:59:14.129181 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54eb1f9-7f2e-441e-a735-fea60e851175 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:59:14.141217 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54eb1f9-7f2e-441e-a735-fea60e851175 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:59:14.146051 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7199de-5a16-497b-a65a-998dbc539edf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:59:14.146699 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7199de-5a16-497b-a65a-998dbc539edf admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:14.147004 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7199de-5a16-497b-a65a-998dbc539edf admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:14.147268 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7199de-5a16-497b-a65a-998dbc539edf admin admin] PATH_INFO: `/v3/projects/dc7b27c40a734bbcb44abaa5590f0df8/users/cf2c17b1a3504773866f3046b4513ed0/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:14.162499 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e7199de-5a16-497b-a65a-998dbc539edf admin admin] RBAC: Authorizing `identity:create_grant(project_id=dc7b27c40a734bbcb44abaa5590f0df8, user_id=cf2c17b1a3504773866f3046b4513ed0, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:59:14.165121 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e7199de-5a16-497b-a65a-998dbc539edf admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:59:14.178338 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1210/2407] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:59:14 2022] PUT /identity/v3/projects/dc7b27c40a734bbcb44abaa5590f0df8/users/cf2c17b1a3504773866f3046b4513ed0/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:59:14.188953 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109faa9d-2704-4b6a-a80a-51ce60ca6f55 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:14.189233 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109faa9d-2704-4b6a-a80a-51ce60ca6f55 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:14.189541 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109faa9d-2704-4b6a-a80a-51ce60ca6f55 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:14.218163 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-109faa9d-2704-4b6a-a80a-51ce60ca6f55 None None] MFA Rules not processed for user `cf2c17b1a3504773866f3046b4513ed0`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:59:14.220008 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-109faa9d-2704-4b6a-a80a-51ce60ca6f55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:59:14.305776 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1198/2408] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:59:14 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:59:14.318162 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7199de-5a16-497b-a65a-998dbc539edf admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:59:14.320185 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-3e7199de-5a16-497b-a65a-998dbc539edf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:59:14.370998 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:59:14.384400 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7199de-5a16-497b-a65a-998dbc539edf admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:59:14.388501 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3baa616c-5933-40f7-af9c-b521003bd128 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:59:14.389152 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3baa616c-5933-40f7-af9c-b521003bd128 service cinder] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:14.389437 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3baa616c-5933-40f7-af9c-b521003bd128 service cinder] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:14.389695 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3baa616c-5933-40f7-af9c-b521003bd128 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:14.400240 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3baa616c-5933-40f7-af9c-b521003bd128 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:59:14.401696 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3baa616c-5933-40f7-af9c-b521003bd128 service cinder] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:59:14.411167 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1211/2409] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:59:14 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:59:17.301605 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109faa9d-2704-4b6a-a80a-51ce60ca6f55 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:59:17.303184 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-109faa9d-2704-4b6a-a80a-51ce60ca6f55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:59:17.363675 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109faa9d-2704-4b6a-a80a-51ce60ca6f55 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:59:17.365768 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39adafca-b525-4813-9159-9a8bf6d40ab0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:59:17.366383 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39adafca-b525-4813-9159-9a8bf6d40ab0 service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:17.366662 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39adafca-b525-4813-9159-9a8bf6d40ab0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:17.366926 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39adafca-b525-4813-9159-9a8bf6d40ab0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:17.368734 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-39adafca-b525-4813-9159-9a8bf6d40ab0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:59:17.421874 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:59:17.444310 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39adafca-b525-4813-9159-9a8bf6d40ab0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:59:17.446302 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39adafca-b525-4813-9159-9a8bf6d40ab0 service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:59:17.457720 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1199/2410] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:59:17 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 157 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:59:24.407214 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=6 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:59:25.011943 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3baa616c-5933-40f7-af9c-b521003bd128 service cinder] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:59:25.075556 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3baa616c-5933-40f7-af9c-b521003bd128 service cinder] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:59:25.080220 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60b9a65-faf5-40f5-a6bc-a20731e8e6fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:59:25.080848 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60b9a65-faf5-40f5-a6bc-a20731e8e6fb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:25.081122 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60b9a65-faf5-40f5-a6bc-a20731e8e6fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:25.081425 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60b9a65-faf5-40f5-a6bc-a20731e8e6fb admin admin] PATH_INFO: `/v3/users/cf2c17b1a3504773866f3046b4513ed0` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:25.082587 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a60b9a65-faf5-40f5-a6bc-a20731e8e6fb admin admin] RBAC: Authorizing `identity:delete_user(user_id=cf2c17b1a3504773866f3046b4513ed0)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:59:25.084494 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a60b9a65-faf5-40f5-a6bc-a20731e8e6fb admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:59:25.132232 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-a60b9a65-faf5-40f5-a6bc-a20731e8e6fb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cf2c17b1a3504773866f3046b4513ed0'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:25.132905 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-a60b9a65-faf5-40f5-a6bc-a20731e8e6fb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cf2c17b1a3504773866f3046b4513ed0'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:25.140997 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-a60b9a65-faf5-40f5-a6bc-a20731e8e6fb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cf2c17b1a3504773866f3046b4513ed0'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:25.141780 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-a60b9a65-faf5-40f5-a6bc-a20731e8e6fb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cf2c17b1a3504773866f3046b4513ed0'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:25.148835 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-a60b9a65-faf5-40f5-a6bc-a20731e8e6fb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cf2c17b1a3504773866f3046b4513ed0'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:25.165854 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1212/2411] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:59:25 2022] DELETE /identity/v3/users/cf2c17b1a3504773866f3046b4513ed0 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:59:25.170377 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d719d7-c177-46cd-ad2e-4f49d211cdf9 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:25.170841 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d719d7-c177-46cd-ad2e-4f49d211cdf9 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:25.170951 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d719d7-c177-46cd-ad2e-4f49d211cdf9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:25.190970 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-39d719d7-c177-46cd-ad2e-4f49d211cdf9 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:59:25.192089 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-39d719d7-c177-46cd-ad2e-4f49d211cdf9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:59:25.255285 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1200/2412] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:59:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 00:59:25.265532 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60b9a65-faf5-40f5-a6bc-a20731e8e6fb admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:59:25.266927 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-a60b9a65-faf5-40f5-a6bc-a20731e8e6fb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:59:25.326999 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60b9a65-faf5-40f5-a6bc-a20731e8e6fb admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:59:25.331132 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33014362-9a66-457c-92fa-5c6ff2197ee1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:59:25.331764 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33014362-9a66-457c-92fa-5c6ff2197ee1 service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:25.332065 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33014362-9a66-457c-92fa-5c6ff2197ee1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:25.332331 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33014362-9a66-457c-92fa-5c6ff2197ee1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:25.342038 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33014362-9a66-457c-92fa-5c6ff2197ee1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:59:25.343472 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33014362-9a66-457c-92fa-5c6ff2197ee1 service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:59:25.353426 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1213/2413] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:59:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:59:25.634845 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d719d7-c177-46cd-ad2e-4f49d211cdf9 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:59:25.636436 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-39d719d7-c177-46cd-ad2e-4f49d211cdf9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:59:25.649573 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d719d7-c177-46cd-ad2e-4f49d211cdf9 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:59:25.654363 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8ff08d0-b4b7-47af-bca3-b1fb6a877def admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:59:25.655082 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ff08d0-b4b7-47af-bca3-b1fb6a877def admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:25.655357 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ff08d0-b4b7-47af-bca3-b1fb6a877def admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:25.656124 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ff08d0-b4b7-47af-bca3-b1fb6a877def admin admin] PATH_INFO: `/v3/projects/dc7b27c40a734bbcb44abaa5590f0df8` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:25.657306 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8ff08d0-b4b7-47af-bca3-b1fb6a877def admin admin] RBAC: Authorizing `identity:delete_project(project_id=dc7b27c40a734bbcb44abaa5590f0df8)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:59:25.659220 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8ff08d0-b4b7-47af-bca3-b1fb6a877def admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:59:25.708292 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-d8ff08d0-b4b7-47af-bca3-b1fb6a877def admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dc7b27c40a734bbcb44abaa5590f0df8', 'request_id': 'req-d8ff08d0-b4b7-47af-bca3-b1fb6a877def'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:25.715557 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-d8ff08d0-b4b7-47af-bca3-b1fb6a877def admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dc7b27c40a734bbcb44abaa5590f0df8', 'request_id': 'req-d8ff08d0-b4b7-47af-bca3-b1fb6a877def'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:25.721845 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-d8ff08d0-b4b7-47af-bca3-b1fb6a877def admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dc7b27c40a734bbcb44abaa5590f0df8', 'request_id': 'req-d8ff08d0-b4b7-47af-bca3-b1fb6a877def'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:25.733194 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-d8ff08d0-b4b7-47af-bca3-b1fb6a877def admin admin] The token cache is being invalidate because project dc7b27c40a734bbcb44abaa5590f0df8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81386) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:59:25.733454 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-d8ff08d0-b4b7-47af-bca3-b1fb6a877def admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:25.734854 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-d8ff08d0-b4b7-47af-bca3-b1fb6a877def admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:25.736353 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1201/2414] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:59:25 2022] DELETE /identity/v3/projects/dc7b27c40a734bbcb44abaa5590f0df8 => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:59:25.742103 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bfed85a-ef5c-434c-887e-ff5ba4f6689f None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:25.742603 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bfed85a-ef5c-434c-887e-ff5ba4f6689f None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:25.742738 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bfed85a-ef5c-434c-887e-ff5ba4f6689f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:25.764063 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-6bfed85a-ef5c-434c-887e-ff5ba4f6689f None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:59:25.765441 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-6bfed85a-ef5c-434c-887e-ff5ba4f6689f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:59:25.783422 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:59:25.829129 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1214/2415] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:59:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:59:25.837094 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea206341-42f9-4b3f-bacb-a224efc113ab None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:25.837437 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea206341-42f9-4b3f-bacb-a224efc113ab None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:25.837694 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea206341-42f9-4b3f-bacb-a224efc113ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:25.858485 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-ea206341-42f9-4b3f-bacb-a224efc113ab None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:59:25.860189 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-ea206341-42f9-4b3f-bacb-a224efc113ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:59:25.868459 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1202/2416] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:59:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:59:25.872781 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bfed85a-ef5c-434c-887e-ff5ba4f6689f None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:59:25.883668 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bfed85a-ef5c-434c-887e-ff5ba4f6689f None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:59:25.888329 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d30a42e-809f-4708-a054-2b94e0bcb82c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:59:25.888943 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d30a42e-809f-4708-a054-2b94e0bcb82c admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:25.889365 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d30a42e-809f-4708-a054-2b94e0bcb82c admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:25.889640 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d30a42e-809f-4708-a054-2b94e0bcb82c admin admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:25.891153 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d30a42e-809f-4708-a054-2b94e0bcb82c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 00:59:25.891457 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d30a42e-809f-4708-a054-2b94e0bcb82c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:59:25.893059 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d30a42e-809f-4708-a054-2b94e0bcb82c admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:59:25.904609 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1215/2417] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 00:59:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 00:59:25.909604 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea206341-42f9-4b3f-bacb-a224efc113ab None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:59:25.919270 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea206341-42f9-4b3f-bacb-a224efc113ab None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:59:25.924905 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11967fe6-cca3-4a8b-8e0b-e9bcd7d47089 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:59:25.925535 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11967fe6-cca3-4a8b-8e0b-e9bcd7d47089 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:25.925804 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11967fe6-cca3-4a8b-8e0b-e9bcd7d47089 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:25.926067 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11967fe6-cca3-4a8b-8e0b-e9bcd7d47089 admin admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:25.926684 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11967fe6-cca3-4a8b-8e0b-e9bcd7d47089 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:59:25.928575 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11967fe6-cca3-4a8b-8e0b-e9bcd7d47089 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:59:25.957638 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1203/2418] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 00:59:25 2022] POST /identity/v3/projects => generated 391 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:59:25.961813 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d30a42e-809f-4708-a054-2b94e0bcb82c admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:59:25.972445 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d30a42e-809f-4708-a054-2b94e0bcb82c admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:59:26.031992 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea742c0c-1e90-4765-84fb-8245b61e8440 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:59:26.032597 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea742c0c-1e90-4765-84fb-8245b61e8440 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:26.032905 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea742c0c-1e90-4765-84fb-8245b61e8440 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:26.033238 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea742c0c-1e90-4765-84fb-8245b61e8440 admin admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:26.033956 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea742c0c-1e90-4765-84fb-8245b61e8440 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:59:26.035888 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea742c0c-1e90-4765-84fb-8245b61e8440 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:59:26.057112 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1216/2419] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 00:59:25 2022] POST /identity/v3/users => generated 296 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 00:59:26.061050 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11967fe6-cca3-4a8b-8e0b-e9bcd7d47089 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:59:26.071813 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11967fe6-cca3-4a8b-8e0b-e9bcd7d47089 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:59:26.076580 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f6dd219-cc2d-40ef-853d-b16b9a888495 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:59:26.077178 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6dd219-cc2d-40ef-853d-b16b9a888495 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:26.077478 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6dd219-cc2d-40ef-853d-b16b9a888495 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:26.077739 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6dd219-cc2d-40ef-853d-b16b9a888495 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:26.078382 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f6dd219-cc2d-40ef-853d-b16b9a888495 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:59:26.079959 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f6dd219-cc2d-40ef-853d-b16b9a888495 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:59:26.091423 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1204/2420] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:59:26 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:59:26.096412 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea742c0c-1e90-4765-84fb-8245b61e8440 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:59:26.106519 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea742c0c-1e90-4765-84fb-8245b61e8440 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:59:26.112118 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c73e757-7959-48dc-ad4b-39c2ee0cafec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:59:26.112737 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c73e757-7959-48dc-ad4b-39c2ee0cafec admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:26.113009 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c73e757-7959-48dc-ad4b-39c2ee0cafec admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:26.113298 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c73e757-7959-48dc-ad4b-39c2ee0cafec admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:26.113938 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c73e757-7959-48dc-ad4b-39c2ee0cafec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:59:26.115507 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c73e757-7959-48dc-ad4b-39c2ee0cafec admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:59:26.127965 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1217/2421] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 00:59:26 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 00:59:26.132115 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f6dd219-cc2d-40ef-853d-b16b9a888495 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:59:26.142869 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f6dd219-cc2d-40ef-853d-b16b9a888495 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:59:26.147632 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf68efa-ac5c-4a3a-9b15-3c825b67e1f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:59:26.148270 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf68efa-ac5c-4a3a-9b15-3c825b67e1f5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:26.148547 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf68efa-ac5c-4a3a-9b15-3c825b67e1f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:26.148818 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf68efa-ac5c-4a3a-9b15-3c825b67e1f5 admin admin] PATH_INFO: `/v3/projects/28dbef675166415687c9840b5ac605f1/users/94c1705b7f0c4b0cacd4d16fe2853478/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:26.164927 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf68efa-ac5c-4a3a-9b15-3c825b67e1f5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=28dbef675166415687c9840b5ac605f1, user_id=94c1705b7f0c4b0cacd4d16fe2853478, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:59:26.167753 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf68efa-ac5c-4a3a-9b15-3c825b67e1f5 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:59:26.176649 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:59:26.181656 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1205/2422] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 00:59:26 2022] PUT /identity/v3/projects/28dbef675166415687c9840b5ac605f1/users/94c1705b7f0c4b0cacd4d16fe2853478/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:59:26.192619 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad2c150-0ab9-44c4-9df3-15530a561cfa None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:26.192902 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad2c150-0ab9-44c4-9df3-15530a561cfa None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:26.193174 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad2c150-0ab9-44c4-9df3-15530a561cfa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:26.221171 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-2ad2c150-0ab9-44c4-9df3-15530a561cfa None None] MFA Rules not processed for user `94c1705b7f0c4b0cacd4d16fe2853478`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:59:26.222344 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-2ad2c150-0ab9-44c4-9df3-15530a561cfa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:59:26.286899 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1218/2423] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:59:26 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:59:26.298169 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf68efa-ac5c-4a3a-9b15-3c825b67e1f5 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:59:26.299928 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-6bf68efa-ac5c-4a3a-9b15-3c825b67e1f5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:59:26.362407 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf68efa-ac5c-4a3a-9b15-3c825b67e1f5 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:59:26.366479 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-032f2992-4d4f-439e-957d-b3927ca01e7e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:59:26.367089 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032f2992-4d4f-439e-957d-b3927ca01e7e service cinder] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:26.367364 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032f2992-4d4f-439e-957d-b3927ca01e7e service cinder] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:26.367628 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032f2992-4d4f-439e-957d-b3927ca01e7e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:26.376698 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-032f2992-4d4f-439e-957d-b3927ca01e7e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:59:26.378226 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-032f2992-4d4f-439e-957d-b3927ca01e7e service cinder] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:59:26.388109 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1206/2424] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:59:26 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:59:35.439935 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89d302b-1f58-4f8d-8f70-6b7f7a97efb7 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:35.439935 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89d302b-1f58-4f8d-8f70-6b7f7a97efb7 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:35.440411 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89d302b-1f58-4f8d-8f70-6b7f7a97efb7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:35.463509 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-b89d302b-1f58-4f8d-8f70-6b7f7a97efb7 None None] MFA Rules not processed for user `3f20ffd8df994c1db1b1e7248db5b5a3`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:59:35.464735 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-b89d302b-1f58-4f8d-8f70-6b7f7a97efb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:59:35.480249 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:59:35.529798 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1219/2425] 10.222.0.52 () {58 vars in 967 bytes} [Sun Sep 25 00:59:35 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:59:35.557074 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-032f2992-4d4f-439e-957d-b3927ca01e7e service cinder] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:59:35.559104 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-032f2992-4d4f-439e-957d-b3927ca01e7e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:59:35.628578 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-032f2992-4d4f-439e-957d-b3927ca01e7e service cinder] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:59:35.633596 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a1ef5c-1a2c-472e-8061-7bed73f543ee service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da606b3906dd4171bd76fd281e03e258', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:59:35.634224 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a1ef5c-1a2c-472e-8061-7bed73f543ee service swift] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:35.634495 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a1ef5c-1a2c-472e-8061-7bed73f543ee service swift] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:35.634758 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a1ef5c-1a2c-472e-8061-7bed73f543ee service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:35.643607 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a1ef5c-1a2c-472e-8061-7bed73f543ee service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:59:35.645165 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a1ef5c-1a2c-472e-8061-7bed73f543ee service swift] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:59:35.653070 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1207/2426] 10.222.0.52 () {60 vars in 1384 bytes} [Sun Sep 25 00:59:35 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 25 00:59:45.522656 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=17 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:59:45.649990 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=58 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:59:45.955941 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b89d302b-1f58-4f8d-8f70-6b7f7a97efb7 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:59:46.022597 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b89d302b-1f58-4f8d-8f70-6b7f7a97efb7 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:59:46.027375 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9e11b0-2488-4de9-85ca-07e307ef6326 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:59:46.028047 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9e11b0-2488-4de9-85ca-07e307ef6326 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:46.028358 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9e11b0-2488-4de9-85ca-07e307ef6326 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:46.028662 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9e11b0-2488-4de9-85ca-07e307ef6326 admin admin] PATH_INFO: `/v3/users/94c1705b7f0c4b0cacd4d16fe2853478` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:46.029931 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a9e11b0-2488-4de9-85ca-07e307ef6326 admin admin] RBAC: Authorizing `identity:delete_user(user_id=94c1705b7f0c4b0cacd4d16fe2853478)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:59:46.032212 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a9e11b0-2488-4de9-85ca-07e307ef6326 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:59:46.102550 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-0a9e11b0-2488-4de9-85ca-07e307ef6326 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '94c1705b7f0c4b0cacd4d16fe2853478'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:46.103354 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-0a9e11b0-2488-4de9-85ca-07e307ef6326 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '94c1705b7f0c4b0cacd4d16fe2853478'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:46.112867 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-0a9e11b0-2488-4de9-85ca-07e307ef6326 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '94c1705b7f0c4b0cacd4d16fe2853478'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:46.113673 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-0a9e11b0-2488-4de9-85ca-07e307ef6326 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '94c1705b7f0c4b0cacd4d16fe2853478'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:46.121722 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-0a9e11b0-2488-4de9-85ca-07e307ef6326 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '94c1705b7f0c4b0cacd4d16fe2853478'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:46.142557 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1220/2427] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 00:59:45 2022] DELETE /identity/v3/users/94c1705b7f0c4b0cacd4d16fe2853478 => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:59:46.147573 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-951203d1-05cc-44b3-b977-6c8f07af741c None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:46.148082 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-951203d1-05cc-44b3-b977-6c8f07af741c None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:46.148334 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-951203d1-05cc-44b3-b977-6c8f07af741c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:46.169848 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-951203d1-05cc-44b3-b977-6c8f07af741c None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:59:46.171495 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-951203d1-05cc-44b3-b977-6c8f07af741c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:59:46.246879 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1208/2428] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:59:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:59:46.257701 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9e11b0-2488-4de9-85ca-07e307ef6326 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:59:46.259298 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-0a9e11b0-2488-4de9-85ca-07e307ef6326 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:59:46.329811 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9e11b0-2488-4de9-85ca-07e307ef6326 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:59:46.334184 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db31dfec-d883-4abe-bcba-f92985251025 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:59:46.334925 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db31dfec-d883-4abe-bcba-f92985251025 service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:46.335120 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db31dfec-d883-4abe-bcba-f92985251025 service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:46.335386 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db31dfec-d883-4abe-bcba-f92985251025 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:46.346738 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db31dfec-d883-4abe-bcba-f92985251025 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:59:46.348219 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db31dfec-d883-4abe-bcba-f92985251025 service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:59:46.360002 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1221/2429] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 00:59:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 00:59:46.709853 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-951203d1-05cc-44b3-b977-6c8f07af741c None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 00:59:46.711538 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-951203d1-05cc-44b3-b977-6c8f07af741c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:59:46.728095 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-951203d1-05cc-44b3-b977-6c8f07af741c None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 00:59:46.733166 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cce3971-2006-45c9-8e98-dabcf10ea3c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 00:59:46.734223 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cce3971-2006-45c9-8e98-dabcf10ea3c6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:46.734634 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cce3971-2006-45c9-8e98-dabcf10ea3c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:46.735223 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cce3971-2006-45c9-8e98-dabcf10ea3c6 admin admin] PATH_INFO: `/v3/projects/28dbef675166415687c9840b5ac605f1` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:46.736275 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cce3971-2006-45c9-8e98-dabcf10ea3c6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=28dbef675166415687c9840b5ac605f1)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 00:59:46.738463 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cce3971-2006-45c9-8e98-dabcf10ea3c6 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 00:59:46.789049 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-1cce3971-2006-45c9-8e98-dabcf10ea3c6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '28dbef675166415687c9840b5ac605f1', 'request_id': 'req-1cce3971-2006-45c9-8e98-dabcf10ea3c6'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:46.797652 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-1cce3971-2006-45c9-8e98-dabcf10ea3c6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '28dbef675166415687c9840b5ac605f1', 'request_id': 'req-1cce3971-2006-45c9-8e98-dabcf10ea3c6'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:46.803513 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-1cce3971-2006-45c9-8e98-dabcf10ea3c6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '28dbef675166415687c9840b5ac605f1', 'request_id': 'req-1cce3971-2006-45c9-8e98-dabcf10ea3c6'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:46.815218 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-1cce3971-2006-45c9-8e98-dabcf10ea3c6 admin admin] The token cache is being invalidate because project 28dbef675166415687c9840b5ac605f1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81386) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 00:59:46.815495 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-1cce3971-2006-45c9-8e98-dabcf10ea3c6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:46.816148 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-1cce3971-2006-45c9-8e98-dabcf10ea3c6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 00:59:46.817765 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1209/2430] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 00:59:46 2022] DELETE /identity/v3/projects/28dbef675166415687c9840b5ac605f1 => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 00:59:55.547614 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3555ed33-6043-491c-afe3-328ca54a71ee None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 00:59:55.548208 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3555ed33-6043-491c-afe3-328ca54a71ee None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 00:59:55.548208 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3555ed33-6043-491c-afe3-328ca54a71ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 00:59:55.570372 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-3555ed33-6043-491c-afe3-328ca54a71ee None None] MFA Rules not processed for user `96386a16f0b54b7c94d32bd948caf118`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 00:59:55.571591 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-3555ed33-6043-491c-afe3-328ca54a71ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 00:59:55.590800 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 00:59:55.637882 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1222/2431] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 00:59:55 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 00:59:56.812202 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=54,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:00:05.630088 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=15 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:00:06.794106 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19208251-db75-4756-b9a3-15b90c2b140e None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:00:06.795212 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19208251-db75-4756-b9a3-15b90c2b140e None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:00:06.795212 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19208251-db75-4756-b9a3-15b90c2b140e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:00:06.815792 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-19208251-db75-4756-b9a3-15b90c2b140e None None] MFA Rules not processed for user `8eabe2a74a604c47bc2e5c88b960dcb6`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:00:06.817003 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-19208251-db75-4756-b9a3-15b90c2b140e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:00:06.873321 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1210/2432] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:00:06 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:00:16.865639 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=25 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:00:25.415110 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57cbb8c-39a1-4abb-9b41-ab2cc9d543f0 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:00:25.415667 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57cbb8c-39a1-4abb-9b41-ab2cc9d543f0 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:00:25.415860 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57cbb8c-39a1-4abb-9b41-ab2cc9d543f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:00:25.437117 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-d57cbb8c-39a1-4abb-9b41-ab2cc9d543f0 None None] MFA Rules not processed for user `dbe551f8c10c4111b4ad7380a6482564`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:00:25.438348 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-d57cbb8c-39a1-4abb-9b41-ab2cc9d543f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:00:25.496187 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1223/2433] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:00:25 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:00:35.489028 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=25 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:00:51.259702 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19208251-db75-4756-b9a3-15b90c2b140e None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:00:51.262069 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-19208251-db75-4756-b9a3-15b90c2b140e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:00:51.340600 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19208251-db75-4756-b9a3-15b90c2b140e None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:00:51.354527 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d99b9adc-4362-4b12-baeb-3d8eaf0a3fa7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:00:51.356140 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d99b9adc-4362-4b12-baeb-3d8eaf0a3fa7 service cinder] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:00:51.356628 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d99b9adc-4362-4b12-baeb-3d8eaf0a3fa7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:00:51.357093 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d99b9adc-4362-4b12-baeb-3d8eaf0a3fa7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:00:51.360430 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-d99b9adc-4362-4b12-baeb-3d8eaf0a3fa7 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:00:51.465968 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d99b9adc-4362-4b12-baeb-3d8eaf0a3fa7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:00:51.467512 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d99b9adc-4362-4b12-baeb-3d8eaf0a3fa7 service cinder] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:00:51.484467 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1211/2434] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:00:51 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 225 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:00:54.252480 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d57cbb8c-39a1-4abb-9b41-ab2cc9d543f0 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:00:54.254160 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-d57cbb8c-39a1-4abb-9b41-ab2cc9d543f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:00:54.327273 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d57cbb8c-39a1-4abb-9b41-ab2cc9d543f0 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:00:54.332824 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4143e2c-825e-4bcf-b5da-e97fb18e3870 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:00:54.333896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4143e2c-825e-4bcf-b5da-e97fb18e3870 service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:00:54.334419 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4143e2c-825e-4bcf-b5da-e97fb18e3870 service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:00:54.334915 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4143e2c-825e-4bcf-b5da-e97fb18e3870 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:00:54.338034 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-b4143e2c-825e-4bcf-b5da-e97fb18e3870 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:00:54.360457 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4143e2c-825e-4bcf-b5da-e97fb18e3870 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:00:54.362734 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4143e2c-825e-4bcf-b5da-e97fb18e3870 service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:00:54.378597 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1224/2435] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:00:54 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 01:00:54.547822 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d99b9adc-4362-4b12-baeb-3d8eaf0a3fa7 service cinder] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:00:54.549758 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-d99b9adc-4362-4b12-baeb-3d8eaf0a3fa7 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:00:54.566382 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d99b9adc-4362-4b12-baeb-3d8eaf0a3fa7 service cinder] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:00:54.572386 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b592eaa-00c4-43f1-8a92-2c04c773c40a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f142988a4dd14a5f96eacea45c046f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:00:54.573008 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b592eaa-00c4-43f1-8a92-2c04c773c40a service nova] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:00:54.573845 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b592eaa-00c4-43f1-8a92-2c04c773c40a service nova] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:00:54.574892 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b592eaa-00c4-43f1-8a92-2c04c773c40a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:00:54.576277 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-9b592eaa-00c4-43f1-8a92-2c04c773c40a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:00:54.588011 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b592eaa-00c4-43f1-8a92-2c04c773c40a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:00:54.590765 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b592eaa-00c4-43f1-8a92-2c04c773c40a service nova] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:00:54.601749 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1212/2436] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:00:54 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 55 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:00:56.776242 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4143e2c-825e-4bcf-b5da-e97fb18e3870 service neutron] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:00:56.778413 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-b4143e2c-825e-4bcf-b5da-e97fb18e3870 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:00:56.794821 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4143e2c-825e-4bcf-b5da-e97fb18e3870 service neutron] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:00:56.801958 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5519da7e-5534-40d7-bc4e-b4fd1289dd85 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f142988a4dd14a5f96eacea45c046f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:00:56.802512 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5519da7e-5534-40d7-bc4e-b4fd1289dd85 service nova] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:00:56.802920 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5519da7e-5534-40d7-bc4e-b4fd1289dd85 service nova] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:00:56.803292 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5519da7e-5534-40d7-bc4e-b4fd1289dd85 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:00:56.805481 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-5519da7e-5534-40d7-bc4e-b4fd1289dd85 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:00:56.818524 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5519da7e-5534-40d7-bc4e-b4fd1289dd85 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:00:56.820306 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5519da7e-5534-40d7-bc4e-b4fd1289dd85 service nova] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:00:56.834630 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1225/2437] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:00:56 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 59 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:01:04.596859 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=60 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:01:06.826324 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=32 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:01:21.470994 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b592eaa-00c4-43f1-8a92-2c04c773c40a service nova] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:21.492320 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b592eaa-00c4-43f1-8a92-2c04c773c40a service nova] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:21.519868 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acec0a9d-38d9-4a46-b740-8b71741f6c45 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f142988a4dd14a5f96eacea45c046f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:21.522381 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acec0a9d-38d9-4a46-b740-8b71741f6c45 service nova] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:21.522809 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acec0a9d-38d9-4a46-b740-8b71741f6c45 service nova] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:21.523088 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acec0a9d-38d9-4a46-b740-8b71741f6c45 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:21.525974 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-acec0a9d-38d9-4a46-b740-8b71741f6c45 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:01:21.551643 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acec0a9d-38d9-4a46-b740-8b71741f6c45 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:21.555004 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acec0a9d-38d9-4a46-b740-8b71741f6c45 service nova] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:21.574994 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1213/2438] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:01:21 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 01:01:22.266354 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5519da7e-5534-40d7-bc4e-b4fd1289dd85 service nova] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:22.286529 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5519da7e-5534-40d7-bc4e-b4fd1289dd85 service nova] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:22.294250 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64fbfce3-3566-42c0-8647-c51034f319c6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:22.295889 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64fbfce3-3566-42c0-8647-c51034f319c6 service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:22.296790 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64fbfce3-3566-42c0-8647-c51034f319c6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:22.297639 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64fbfce3-3566-42c0-8647-c51034f319c6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:22.301606 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-64fbfce3-3566-42c0-8647-c51034f319c6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:01:22.347119 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acec0a9d-38d9-4a46-b740-8b71741f6c45 service nova] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:22.383156 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acec0a9d-38d9-4a46-b740-8b71741f6c45 service nova] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:22.393942 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c279ed0-ed02-4064-8dc4-9c070ccdd9ec service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f142988a4dd14a5f96eacea45c046f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:22.394715 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c279ed0-ed02-4064-8dc4-9c070ccdd9ec service nova] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:22.395146 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c279ed0-ed02-4064-8dc4-9c070ccdd9ec service nova] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:22.395808 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c279ed0-ed02-4064-8dc4-9c070ccdd9ec service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:22.399784 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-8c279ed0-ed02-4064-8dc4-9c070ccdd9ec service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:01:22.432360 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c279ed0-ed02-4064-8dc4-9c070ccdd9ec service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:22.437736 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c279ed0-ed02-4064-8dc4-9c070ccdd9ec service nova] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:22.487915 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1214/2439] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:01:22 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:01:22.521434 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64fbfce3-3566-42c0-8647-c51034f319c6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:22.526615 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64fbfce3-3566-42c0-8647-c51034f319c6 service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:22.547749 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1226/2440] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:01:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 283 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:01:22.965878 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806def9f-5a72-4085-b316-79b2f692958e None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:22.966429 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806def9f-5a72-4085-b316-79b2f692958e None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:22.966979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806def9f-5a72-4085-b316-79b2f692958e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:23.046042 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-806def9f-5a72-4085-b316-79b2f692958e None None] MFA Rules not processed for user `3f20ffd8df994c1db1b1e7248db5b5a3`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:01:23.050498 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-806def9f-5a72-4085-b316-79b2f692958e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:01:23.281123 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1215/2441] 10.222.0.52 () {58 vars in 967 bytes} [Sun Sep 25 01:01:22 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 320 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 01:01:23.389003 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64fbfce3-3566-42c0-8647-c51034f319c6 service neutron] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:23.392732 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-64fbfce3-3566-42c0-8647-c51034f319c6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:01:23.565150 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64fbfce3-3566-42c0-8647-c51034f319c6 service neutron] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:23.582834 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae09a27-478d-4316-a311-054c25102dcf service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da606b3906dd4171bd76fd281e03e258', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:23.585481 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae09a27-478d-4316-a311-054c25102dcf service swift] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:23.589147 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae09a27-478d-4316-a311-054c25102dcf service swift] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:23.590845 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae09a27-478d-4316-a311-054c25102dcf service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:23.612418 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ae09a27-478d-4316-a311-054c25102dcf service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:23.614819 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ae09a27-478d-4316-a311-054c25102dcf service swift] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:23.655460 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1227/2442] 10.222.0.52 () {60 vars in 1384 bytes} [Sun Sep 25 01:01:23 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 265 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 25 01:01:25.331105 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-806def9f-5a72-4085-b316-79b2f692958e None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:25.362152 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-806def9f-5a72-4085-b316-79b2f692958e None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:25.383808 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3412cede-afea-4512-9d86-a96de04df9c6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f142988a4dd14a5f96eacea45c046f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:25.386872 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3412cede-afea-4512-9d86-a96de04df9c6 service nova] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:25.387545 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3412cede-afea-4512-9d86-a96de04df9c6 service nova] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:25.387922 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3412cede-afea-4512-9d86-a96de04df9c6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:25.390817 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-3412cede-afea-4512-9d86-a96de04df9c6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:01:25.409673 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3412cede-afea-4512-9d86-a96de04df9c6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:25.411493 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3412cede-afea-4512-9d86-a96de04df9c6 service nova] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:25.425391 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1216/2443] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:01:25 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:01:25.481697 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0413f2-94b9-4b70-8ea4-49fe37841dec None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:25.482553 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0413f2-94b9-4b70-8ea4-49fe37841dec None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:25.483005 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0413f2-94b9-4b70-8ea4-49fe37841dec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:25.538430 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-5e0413f2-94b9-4b70-8ea4-49fe37841dec None None] MFA Rules not processed for user `c19b10ebc31244dbb2b1539d94ac281d`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:01:25.540040 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-5e0413f2-94b9-4b70-8ea4-49fe37841dec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:01:25.661236 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1228/2444] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:01:25 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:01:26.639650 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3412cede-afea-4512-9d86-a96de04df9c6 service nova] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:26.659112 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3412cede-afea-4512-9d86-a96de04df9c6 service nova] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:26.668166 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-775ca013-92ab-4456-b002-f4a53926461f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:26.669476 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775ca013-92ab-4456-b002-f4a53926461f service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:26.670176 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775ca013-92ab-4456-b002-f4a53926461f service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:26.670973 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775ca013-92ab-4456-b002-f4a53926461f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:26.674442 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-775ca013-92ab-4456-b002-f4a53926461f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:01:26.705091 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-775ca013-92ab-4456-b002-f4a53926461f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:26.709424 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-775ca013-92ab-4456-b002-f4a53926461f service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:26.748747 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1217/2445] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:01:26 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:01:28.330347 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0413f2-94b9-4b70-8ea4-49fe37841dec None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:28.332640 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-5e0413f2-94b9-4b70-8ea4-49fe37841dec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:01:28.432471 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:01:28.577689 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0413f2-94b9-4b70-8ea4-49fe37841dec None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:28.591771 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb9b28b-89a4-46b5-9abb-6bf2b73c1c0b service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '11a80b55dcd444df901c8bf80f2a2c26', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:28.594333 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb9b28b-89a4-46b5-9abb-6bf2b73c1c0b service placement] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:28.595223 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb9b28b-89a4-46b5-9abb-6bf2b73c1c0b service placement] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:28.598641 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb9b28b-89a4-46b5-9abb-6bf2b73c1c0b service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:28.604597 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-1fb9b28b-89a4-46b5-9abb-6bf2b73c1c0b service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:01:28.640300 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb9b28b-89a4-46b5-9abb-6bf2b73c1c0b service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:28.644297 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb9b28b-89a4-46b5-9abb-6bf2b73c1c0b service placement] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:28.681364 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1229/2446] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:01:28 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 352 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 01:01:30.005410 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0e96d7-f21e-4d32-a13c-2d1294718ad4 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:30.005410 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0e96d7-f21e-4d32-a13c-2d1294718ad4 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:30.005838 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0e96d7-f21e-4d32-a13c-2d1294718ad4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:30.048024 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-1c0e96d7-f21e-4d32-a13c-2d1294718ad4 None None] MFA Rules not processed for user `96386a16f0b54b7c94d32bd948caf118`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:01:30.052148 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-1c0e96d7-f21e-4d32-a13c-2d1294718ad4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:01:30.068858 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1218/2447] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:01:30 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 65 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:01:30.895004 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb9b28b-89a4-46b5-9abb-6bf2b73c1c0b service placement] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:30.899416 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-1fb9b28b-89a4-46b5-9abb-6bf2b73c1c0b service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:01:30.930501 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb9b28b-89a4-46b5-9abb-6bf2b73c1c0b service placement] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:30.941111 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5d122be-963f-4122-b95e-45aaba29eab8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:30.942690 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d122be-963f-4122-b95e-45aaba29eab8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:30.943324 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d122be-963f-4122-b95e-45aaba29eab8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:30.944203 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d122be-963f-4122-b95e-45aaba29eab8 admin admin] PATH_INFO: `/v3/users/8eabe2a74a604c47bc2e5c88b960dcb6` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:30.948373 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5d122be-963f-4122-b95e-45aaba29eab8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8eabe2a74a604c47bc2e5c88b960dcb6)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:30.951309 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5d122be-963f-4122-b95e-45aaba29eab8 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:31.044619 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-c5d122be-963f-4122-b95e-45aaba29eab8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8eabe2a74a604c47bc2e5c88b960dcb6'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:31.045973 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-c5d122be-963f-4122-b95e-45aaba29eab8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8eabe2a74a604c47bc2e5c88b960dcb6'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:31.061190 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-c5d122be-963f-4122-b95e-45aaba29eab8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8eabe2a74a604c47bc2e5c88b960dcb6'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:31.072619 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-c5d122be-963f-4122-b95e-45aaba29eab8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8eabe2a74a604c47bc2e5c88b960dcb6'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:31.085731 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-c5d122be-963f-4122-b95e-45aaba29eab8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8eabe2a74a604c47bc2e5c88b960dcb6'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:31.123735 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1230/2448] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 01:01:30 2022] DELETE /identity/v3/users/8eabe2a74a604c47bc2e5c88b960dcb6 => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 01:01:31.450265 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c0e96d7-f21e-4d32-a13c-2d1294718ad4 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:31.470582 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c0e96d7-f21e-4d32-a13c-2d1294718ad4 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:31.610892 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f347c1b2-5a97-4f37-ad0e-9028b750bc16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:31.612423 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f347c1b2-5a97-4f37-ad0e-9028b750bc16 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:31.613184 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f347c1b2-5a97-4f37-ad0e-9028b750bc16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:31.614004 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f347c1b2-5a97-4f37-ad0e-9028b750bc16 admin admin] PATH_INFO: `/v3/projects/28568584195d475fbc52d76185e09044` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:31.617851 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f347c1b2-5a97-4f37-ad0e-9028b750bc16 admin admin] RBAC: Authorizing `identity:delete_project(project_id=28568584195d475fbc52d76185e09044)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:31.622737 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f347c1b2-5a97-4f37-ad0e-9028b750bc16 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:31.724273 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:01:31.726000 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-f347c1b2-5a97-4f37-ad0e-9028b750bc16 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '28568584195d475fbc52d76185e09044', 'request_id': 'req-f347c1b2-5a97-4f37-ad0e-9028b750bc16'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:31.742962 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-f347c1b2-5a97-4f37-ad0e-9028b750bc16 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '28568584195d475fbc52d76185e09044', 'request_id': 'req-f347c1b2-5a97-4f37-ad0e-9028b750bc16'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:31.759832 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-f347c1b2-5a97-4f37-ad0e-9028b750bc16 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '28568584195d475fbc52d76185e09044', 'request_id': 'req-f347c1b2-5a97-4f37-ad0e-9028b750bc16'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:31.787950 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-f347c1b2-5a97-4f37-ad0e-9028b750bc16 admin admin] The token cache is being invalidate because project 28568584195d475fbc52d76185e09044 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81386) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 01:01:31.788422 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-f347c1b2-5a97-4f37-ad0e-9028b750bc16 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:31.789419 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-f347c1b2-5a97-4f37-ad0e-9028b750bc16 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:31.792230 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1219/2449] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 01:01:31 2022] DELETE /identity/v3/projects/28568584195d475fbc52d76185e09044 => generated 0 bytes in 343 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:01:31.810578 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe94bef-49f0-4842-a183-0fc760af26b1 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:31.816072 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe94bef-49f0-4842-a183-0fc760af26b1 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:31.816959 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe94bef-49f0-4842-a183-0fc760af26b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:31.883700 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-8fe94bef-49f0-4842-a183-0fc760af26b1 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:01:31.887164 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-8fe94bef-49f0-4842-a183-0fc760af26b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:01:32.158630 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1231/2450] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:01:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 351 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:01:32.163068 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151c1d32-e345-46f4-a95c-c03cedc20eda None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:32.165166 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151c1d32-e345-46f4-a95c-c03cedc20eda None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:32.166681 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151c1d32-e345-46f4-a95c-c03cedc20eda None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:32.220574 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-151c1d32-e345-46f4-a95c-c03cedc20eda None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:01:32.223247 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-151c1d32-e345-46f4-a95c-c03cedc20eda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:01:32.263454 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1220/2451] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:01:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 01:01:32.280170 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fe94bef-49f0-4842-a183-0fc760af26b1 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:32.314321 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fe94bef-49f0-4842-a183-0fc760af26b1 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:32.337904 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e42ba67-d197-4448-9d6c-2844fd898c69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:32.339472 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e42ba67-d197-4448-9d6c-2844fd898c69 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:32.340204 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e42ba67-d197-4448-9d6c-2844fd898c69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:32.340926 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e42ba67-d197-4448-9d6c-2844fd898c69 admin admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:32.342408 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e42ba67-d197-4448-9d6c-2844fd898c69 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 01:01:32.343274 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e42ba67-d197-4448-9d6c-2844fd898c69 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:32.347673 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e42ba67-d197-4448-9d6c-2844fd898c69 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:32.375288 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1232/2452] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 01:01:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 01:01:32.382306 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-151c1d32-e345-46f4-a95c-c03cedc20eda None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:32.408894 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-151c1d32-e345-46f4-a95c-c03cedc20eda None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:32.435207 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0832050-fa1b-4348-b92c-3754d2bbc190 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:32.436402 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0832050-fa1b-4348-b92c-3754d2bbc190 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:32.436830 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0832050-fa1b-4348-b92c-3754d2bbc190 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:32.437358 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0832050-fa1b-4348-b92c-3754d2bbc190 admin admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:32.438528 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0832050-fa1b-4348-b92c-3754d2bbc190 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:32.444023 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0832050-fa1b-4348-b92c-3754d2bbc190 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:32.512091 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1221/2453] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 01:01:32 2022] POST /identity/v3/projects => generated 405 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 01:01:32.516930 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e42ba67-d197-4448-9d6c-2844fd898c69 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:32.536428 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e42ba67-d197-4448-9d6c-2844fd898c69 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:32.558046 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:01:32.662299 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a72ecd-16cd-4a40-bccb-5f4e7604a90e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:32.663696 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a72ecd-16cd-4a40-bccb-5f4e7604a90e admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:32.664434 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a72ecd-16cd-4a40-bccb-5f4e7604a90e admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:32.665098 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a72ecd-16cd-4a40-bccb-5f4e7604a90e admin admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:32.667096 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28a72ecd-16cd-4a40-bccb-5f4e7604a90e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:32.673935 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28a72ecd-16cd-4a40-bccb-5f4e7604a90e admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:32.719997 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1233/2454] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 01:01:32 2022] POST /identity/v3/users => generated 303 bytes in 203 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 01:01:32.726850 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0832050-fa1b-4348-b92c-3754d2bbc190 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:32.754680 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0832050-fa1b-4348-b92c-3754d2bbc190 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:32.765578 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bcec1b9-358a-40f0-8626-5b65a9cc6b35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:32.766625 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bcec1b9-358a-40f0-8626-5b65a9cc6b35 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:32.767075 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bcec1b9-358a-40f0-8626-5b65a9cc6b35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:32.767546 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bcec1b9-358a-40f0-8626-5b65a9cc6b35 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:32.768749 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bcec1b9-358a-40f0-8626-5b65a9cc6b35 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:32.772398 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bcec1b9-358a-40f0-8626-5b65a9cc6b35 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:32.802529 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1222/2455] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:01:32 2022] GET /identity/v3/roles => generated 1420 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 01:01:32.807130 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a72ecd-16cd-4a40-bccb-5f4e7604a90e admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:32.833532 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a72ecd-16cd-4a40-bccb-5f4e7604a90e admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:32.844368 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bdd5c65-375f-454e-8a98-a3120ee523f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:32.846176 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bdd5c65-375f-454e-8a98-a3120ee523f3 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:32.846176 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bdd5c65-375f-454e-8a98-a3120ee523f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:32.846853 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bdd5c65-375f-454e-8a98-a3120ee523f3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:32.848664 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bdd5c65-375f-454e-8a98-a3120ee523f3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:32.851575 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bdd5c65-375f-454e-8a98-a3120ee523f3 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:32.884523 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1234/2456] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:01:32 2022] GET /identity/v3/roles => generated 1420 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 01:01:32.889633 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bcec1b9-358a-40f0-8626-5b65a9cc6b35 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:32.924369 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bcec1b9-358a-40f0-8626-5b65a9cc6b35 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:32.938169 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044c969b-be3e-421a-b2b8-a16f9618f7d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:32.941173 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044c969b-be3e-421a-b2b8-a16f9618f7d8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:32.942020 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044c969b-be3e-421a-b2b8-a16f9618f7d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:32.943063 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044c969b-be3e-421a-b2b8-a16f9618f7d8 admin admin] PATH_INFO: `/v3/projects/a0010b18914c438d9828a77f9fa2237e/users/a642abaaaa8945d3a6b18f1c8eb08488/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:32.972612 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-044c969b-be3e-421a-b2b8-a16f9618f7d8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a0010b18914c438d9828a77f9fa2237e, user_id=a642abaaaa8945d3a6b18f1c8eb08488, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:32.976110 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-044c969b-be3e-421a-b2b8-a16f9618f7d8 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:33.013669 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1223/2457] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 01:01:32 2022] PUT /identity/v3/projects/a0010b18914c438d9828a77f9fa2237e/users/a642abaaaa8945d3a6b18f1c8eb08488/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:01:33.016550 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31822163-f6ae-4333-b5fb-f522fee5a0f2 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:33.017211 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31822163-f6ae-4333-b5fb-f522fee5a0f2 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:33.017988 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31822163-f6ae-4333-b5fb-f522fee5a0f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:33.064498 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-31822163-f6ae-4333-b5fb-f522fee5a0f2 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:01:33.071100 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-31822163-f6ae-4333-b5fb-f522fee5a0f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:01:33.273090 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1235/2458] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:01:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 259 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:01:33.280905 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044c969b-be3e-421a-b2b8-a16f9618f7d8 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:33.283325 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-044c969b-be3e-421a-b2b8-a16f9618f7d8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:01:33.371445 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044c969b-be3e-421a-b2b8-a16f9618f7d8 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:33.390470 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-593e84f5-2aef-419d-a94b-24ec7315979d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:33.391593 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593e84f5-2aef-419d-a94b-24ec7315979d service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:33.392014 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593e84f5-2aef-419d-a94b-24ec7315979d service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:33.392418 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593e84f5-2aef-419d-a94b-24ec7315979d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:33.407870 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31822163-f6ae-4333-b5fb-f522fee5a0f2 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:33.423547 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-593e84f5-2aef-419d-a94b-24ec7315979d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:33.424426 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31822163-f6ae-4333-b5fb-f522fee5a0f2 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:33.430181 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-593e84f5-2aef-419d-a94b-24ec7315979d service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:33.435843 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5f7095-6f52-43d2-b68b-88d0b341138b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:33.436482 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5f7095-6f52-43d2-b68b-88d0b341138b service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:33.436758 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5f7095-6f52-43d2-b68b-88d0b341138b service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:33.437029 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5f7095-6f52-43d2-b68b-88d0b341138b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:33.447370 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-2f5f7095-6f52-43d2-b68b-88d0b341138b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:01:33.464507 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1224/2459] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:01:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 01:01:33.484443 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f5f7095-6f52-43d2-b68b-88d0b341138b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:33.494325 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f5f7095-6f52-43d2-b68b-88d0b341138b service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:33.520308 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1236/2460] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:01:33 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 01:01:36.076485 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58db3b5a-7b40-4c0d-a2aa-397fc34d9d75 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:36.077477 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58db3b5a-7b40-4c0d-a2aa-397fc34d9d75 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:36.077676 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58db3b5a-7b40-4c0d-a2aa-397fc34d9d75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:36.236604 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-58db3b5a-7b40-4c0d-a2aa-397fc34d9d75 None None] MFA Rules not processed for user `a642abaaaa8945d3a6b18f1c8eb08488`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:01:36.250892 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-58db3b5a-7b40-4c0d-a2aa-397fc34d9d75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:01:36.574427 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1225/2461] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:01:36 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 496 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:01:36.575777 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5f7095-6f52-43d2-b68b-88d0b341138b service neutron] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:36.589041 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5f7095-6f52-43d2-b68b-88d0b341138b service neutron] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:36.595558 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50604166-e46c-477d-ab5d-6beff4e3ff5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:36.596593 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50604166-e46c-477d-ab5d-6beff4e3ff5e admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:36.597033 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50604166-e46c-477d-ab5d-6beff4e3ff5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:36.597492 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50604166-e46c-477d-ab5d-6beff4e3ff5e admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:36.598441 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50604166-e46c-477d-ab5d-6beff4e3ff5e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:36.600805 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50604166-e46c-477d-ab5d-6beff4e3ff5e admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:36.648421 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1237/2462] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 01:01:36 2022] POST /identity/v3/projects => generated 403 bytes in 73 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 01:01:36.649729 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58db3b5a-7b40-4c0d-a2aa-397fc34d9d75 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:36.664426 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=94,keystone:INSERT=3,keystone:DELETE=3 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:01:36.677241 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58db3b5a-7b40-4c0d-a2aa-397fc34d9d75 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:36.819244 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6adbb249-6e77-4ddf-8878-c23c462c0655 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:36.819996 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6adbb249-6e77-4ddf-8878-c23c462c0655 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:36.820375 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6adbb249-6e77-4ddf-8878-c23c462c0655 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:36.820833 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6adbb249-6e77-4ddf-8878-c23c462c0655 admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:36.821721 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6adbb249-6e77-4ddf-8878-c23c462c0655 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:36.823976 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6adbb249-6e77-4ddf-8878-c23c462c0655 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:36.870089 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1226/2463] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 01:01:36 2022] POST /identity/v3/users => generated 308 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 01:01:36.873711 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50604166-e46c-477d-ab5d-6beff4e3ff5e admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:36.885543 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50604166-e46c-477d-ab5d-6beff4e3ff5e admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:36.892020 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf21c49-2e38-4df9-8e5b-8fb7ca5e6611 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:36.892020 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf21c49-2e38-4df9-8e5b-8fb7ca5e6611 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:36.892507 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf21c49-2e38-4df9-8e5b-8fb7ca5e6611 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:36.892507 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf21c49-2e38-4df9-8e5b-8fb7ca5e6611 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:36.893514 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdf21c49-2e38-4df9-8e5b-8fb7ca5e6611 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:36.896652 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdf21c49-2e38-4df9-8e5b-8fb7ca5e6611 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:36.911780 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1238/2464] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:01:36 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 01:01:36.913967 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6adbb249-6e77-4ddf-8878-c23c462c0655 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:36.954891 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6adbb249-6e77-4ddf-8878-c23c462c0655 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:36.969914 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16baab7e-c6ba-4a73-b897-80536241eb96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:36.972051 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16baab7e-c6ba-4a73-b897-80536241eb96 admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:36.973587 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16baab7e-c6ba-4a73-b897-80536241eb96 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:36.973587 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16baab7e-c6ba-4a73-b897-80536241eb96 admin admin] PATH_INFO: `/v3/projects/72ebfef0864443e598916e212e9811df/users/29d87d3093d44b83850f8128f84f99cd/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:37.048602 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16baab7e-c6ba-4a73-b897-80536241eb96 admin admin] RBAC: Authorizing `identity:create_grant(project_id=72ebfef0864443e598916e212e9811df, user_id=29d87d3093d44b83850f8128f84f99cd, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:37.053019 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16baab7e-c6ba-4a73-b897-80536241eb96 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:37.114966 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf21c49-2e38-4df9-8e5b-8fb7ca5e6611 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:37.125609 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1227/2465] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 01:01:36 2022] PUT /identity/v3/projects/72ebfef0864443e598916e212e9811df/users/29d87d3093d44b83850f8128f84f99cd/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 01:01:37.134459 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf21c49-2e38-4df9-8e5b-8fb7ca5e6611 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:37.244040 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:01:37.270991 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-541266c1-22ca-48d0-a8d0-9c4b3ccefd18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:37.274811 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541266c1-22ca-48d0-a8d0-9c4b3ccefd18 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:37.275519 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541266c1-22ca-48d0-a8d0-9c4b3ccefd18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:37.275955 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541266c1-22ca-48d0-a8d0-9c4b3ccefd18 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:37.276842 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-541266c1-22ca-48d0-a8d0-9c4b3ccefd18 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:37.278763 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-541266c1-22ca-48d0-a8d0-9c4b3ccefd18 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:37.296109 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1239/2466] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:01:37 2022] GET /identity/v3/roles => generated 1420 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 01:01:37.297470 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16baab7e-c6ba-4a73-b897-80536241eb96 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:37.310130 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16baab7e-c6ba-4a73-b897-80536241eb96 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:37.323738 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-675e1f6f-29e0-48c5-bb33-dd11288b6299 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:37.324522 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-675e1f6f-29e0-48c5-bb33-dd11288b6299 admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:37.325167 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-675e1f6f-29e0-48c5-bb33-dd11288b6299 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:37.325786 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-675e1f6f-29e0-48c5-bb33-dd11288b6299 admin admin] PATH_INFO: `/v3/projects/72ebfef0864443e598916e212e9811df/users/29d87d3093d44b83850f8128f84f99cd/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:37.328419 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-675e1f6f-29e0-48c5-bb33-dd11288b6299 admin admin] RBAC: Authorizing `identity:create_grant(project_id=72ebfef0864443e598916e212e9811df, user_id=29d87d3093d44b83850f8128f84f99cd, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:37.331143 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-675e1f6f-29e0-48c5-bb33-dd11288b6299 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:37.345901 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1228/2467] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 01:01:37 2022] PUT /identity/v3/projects/72ebfef0864443e598916e212e9811df/users/29d87d3093d44b83850f8128f84f99cd/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 01:01:38.697506 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-541266c1-22ca-48d0-a8d0-9c4b3ccefd18 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:38.699710 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-541266c1-22ca-48d0-a8d0-9c4b3ccefd18 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:01:38.717047 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-541266c1-22ca-48d0-a8d0-9c4b3ccefd18 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:38.805175 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0d209f-414c-4c77-92f2-c8e2e24ea17c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:38.807059 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0d209f-414c-4c77-92f2-c8e2e24ea17c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:38.807427 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0d209f-414c-4c77-92f2-c8e2e24ea17c admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:38.807674 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0d209f-414c-4c77-92f2-c8e2e24ea17c admin admin] PATH_INFO: `/v3/users/80dd52f1455f44b587716a41e632a155` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:38.809188 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e0d209f-414c-4c77-92f2-c8e2e24ea17c admin admin] RBAC: Authorizing `identity:delete_user(user_id=80dd52f1455f44b587716a41e632a155)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:38.811372 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e0d209f-414c-4c77-92f2-c8e2e24ea17c admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:38.885826 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-4e0d209f-414c-4c77-92f2-c8e2e24ea17c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '80dd52f1455f44b587716a41e632a155'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:38.887599 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-4e0d209f-414c-4c77-92f2-c8e2e24ea17c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '80dd52f1455f44b587716a41e632a155'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:38.898562 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-4e0d209f-414c-4c77-92f2-c8e2e24ea17c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '80dd52f1455f44b587716a41e632a155'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:38.899425 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-4e0d209f-414c-4c77-92f2-c8e2e24ea17c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '80dd52f1455f44b587716a41e632a155'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:38.914404 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-4e0d209f-414c-4c77-92f2-c8e2e24ea17c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '80dd52f1455f44b587716a41e632a155'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:38.940411 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1240/2468] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 01:01:38 2022] DELETE /identity/v3/users/80dd52f1455f44b587716a41e632a155 => generated 0 bytes in 244 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 01:01:39.631164 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-675e1f6f-29e0-48c5-bb33-dd11288b6299 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:39.635347 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-675e1f6f-29e0-48c5-bb33-dd11288b6299 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:01:39.652675 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-675e1f6f-29e0-48c5-bb33-dd11288b6299 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:39.764602 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d6546b-907d-42da-91e8-6209e372d5ae None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:39.765369 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d6546b-907d-42da-91e8-6209e372d5ae None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:39.765369 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d6546b-907d-42da-91e8-6209e372d5ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:39.773967 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb5c1cf-dda0-4060-81d0-8fbc4bc1e037 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:39.774900 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb5c1cf-dda0-4060-81d0-8fbc4bc1e037 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:39.775204 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb5c1cf-dda0-4060-81d0-8fbc4bc1e037 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:39.775640 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb5c1cf-dda0-4060-81d0-8fbc4bc1e037 admin admin] PATH_INFO: `/v3/projects/b161c2e2dd1f42159dc0d71e8216516c` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:39.777996 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bb5c1cf-dda0-4060-81d0-8fbc4bc1e037 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b161c2e2dd1f42159dc0d71e8216516c)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:39.785807 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bb5c1cf-dda0-4060-81d0-8fbc4bc1e037 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:39.830163 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-b1d6546b-907d-42da-91e8-6209e372d5ae None None] MFA Rules not processed for user `29d87d3093d44b83850f8128f84f99cd`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:01:39.831887 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-b1d6546b-907d-42da-91e8-6209e372d5ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:01:39.890746 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-2bb5c1cf-dda0-4060-81d0-8fbc4bc1e037 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b161c2e2dd1f42159dc0d71e8216516c', 'request_id': 'req-2bb5c1cf-dda0-4060-81d0-8fbc4bc1e037'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:39.907493 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-2bb5c1cf-dda0-4060-81d0-8fbc4bc1e037 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b161c2e2dd1f42159dc0d71e8216516c', 'request_id': 'req-2bb5c1cf-dda0-4060-81d0-8fbc4bc1e037'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:39.912364 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=88,keystone:INSERT=6,keystone:DELETE=6 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:01:39.923872 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-2bb5c1cf-dda0-4060-81d0-8fbc4bc1e037 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b161c2e2dd1f42159dc0d71e8216516c', 'request_id': 'req-2bb5c1cf-dda0-4060-81d0-8fbc4bc1e037'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:39.975259 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-2bb5c1cf-dda0-4060-81d0-8fbc4bc1e037 admin admin] The token cache is being invalidate because project b161c2e2dd1f42159dc0d71e8216516c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81386) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 01:01:39.975767 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-2bb5c1cf-dda0-4060-81d0-8fbc4bc1e037 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:39.976898 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-2bb5c1cf-dda0-4060-81d0-8fbc4bc1e037 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:39.979645 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1229/2469] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 01:01:39 2022] DELETE /identity/v3/projects/b161c2e2dd1f42159dc0d71e8216516c => generated 0 bytes in 349 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 01:01:39.981076 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:01:39.987465 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb5c1cf-dda0-4060-81d0-8fbc4bc1e037 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:39.991075 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-2bb5c1cf-dda0-4060-81d0-8fbc4bc1e037 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:01:40.045299 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1241/2470] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:01:39 2022] POST /identity/v3/auth/tokens => generated 3347 bytes in 282 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:01:40.124816 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb5c1cf-dda0-4060-81d0-8fbc4bc1e037 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:40.135002 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c93310ab-1b28-4603-9bf6-0b55c181281f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:40.135728 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93310ab-1b28-4603-9bf6-0b55c181281f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:40.136271 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93310ab-1b28-4603-9bf6-0b55c181281f admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:40.137749 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93310ab-1b28-4603-9bf6-0b55c181281f admin admin] PATH_INFO: `/v3/users/972219920bf74507b005a7c9d10dd6bd` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:40.139571 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c93310ab-1b28-4603-9bf6-0b55c181281f admin admin] RBAC: Authorizing `identity:delete_user(user_id=972219920bf74507b005a7c9d10dd6bd)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:40.145143 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c93310ab-1b28-4603-9bf6-0b55c181281f admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:40.252564 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-c93310ab-1b28-4603-9bf6-0b55c181281f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '972219920bf74507b005a7c9d10dd6bd'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:40.265318 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-c93310ab-1b28-4603-9bf6-0b55c181281f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '972219920bf74507b005a7c9d10dd6bd'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:40.266642 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-c93310ab-1b28-4603-9bf6-0b55c181281f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '972219920bf74507b005a7c9d10dd6bd'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:40.296461 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-c93310ab-1b28-4603-9bf6-0b55c181281f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '972219920bf74507b005a7c9d10dd6bd'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:40.299065 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-c93310ab-1b28-4603-9bf6-0b55c181281f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '972219920bf74507b005a7c9d10dd6bd'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:40.330263 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1230/2471] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 01:01:39 2022] DELETE /identity/v3/users/972219920bf74507b005a7c9d10dd6bd => generated 0 bytes in 342 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 01:01:40.770362 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d6546b-907d-42da-91e8-6209e372d5ae None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:40.787360 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d6546b-907d-42da-91e8-6209e372d5ae None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:40.875237 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e01216-e733-4a5d-aacb-1672246a0ad2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:40.876180 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e01216-e733-4a5d-aacb-1672246a0ad2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:40.876788 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e01216-e733-4a5d-aacb-1672246a0ad2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:40.877516 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e01216-e733-4a5d-aacb-1672246a0ad2 admin admin] PATH_INFO: `/v3/projects/31719d4e28f048bbbf6e307df745e648` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:40.879841 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7e01216-e733-4a5d-aacb-1672246a0ad2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=31719d4e28f048bbbf6e307df745e648)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:40.882077 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7e01216-e733-4a5d-aacb-1672246a0ad2 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:40.948084 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-a7e01216-e733-4a5d-aacb-1672246a0ad2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '31719d4e28f048bbbf6e307df745e648', 'request_id': 'req-a7e01216-e733-4a5d-aacb-1672246a0ad2'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:40.959346 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-a7e01216-e733-4a5d-aacb-1672246a0ad2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '31719d4e28f048bbbf6e307df745e648', 'request_id': 'req-a7e01216-e733-4a5d-aacb-1672246a0ad2'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:40.968184 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-a7e01216-e733-4a5d-aacb-1672246a0ad2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '31719d4e28f048bbbf6e307df745e648', 'request_id': 'req-a7e01216-e733-4a5d-aacb-1672246a0ad2'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:40.982837 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-a7e01216-e733-4a5d-aacb-1672246a0ad2 admin admin] The token cache is being invalidate because project 31719d4e28f048bbbf6e307df745e648 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 01:01:40.983157 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-a7e01216-e733-4a5d-aacb-1672246a0ad2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:40.984124 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-a7e01216-e733-4a5d-aacb-1672246a0ad2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:40.989703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1242/2472] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 01:01:40 2022] DELETE /identity/v3/projects/31719d4e28f048bbbf6e307df745e648 => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:01:40.992847 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5644f97-37f1-4875-ba5b-27fdd053d642 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:40.993864 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5644f97-37f1-4875-ba5b-27fdd053d642 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:40.994297 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5644f97-37f1-4875-ba5b-27fdd053d642 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:41.025788 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-e5644f97-37f1-4875-ba5b-27fdd053d642 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:01:41.026731 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-e5644f97-37f1-4875-ba5b-27fdd053d642 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:01:41.133319 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1231/2473] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:01:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 01:01:41.143513 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed001ff-f575-4271-b31c-e4a364a8bfa6 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:41.143513 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed001ff-f575-4271-b31c-e4a364a8bfa6 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:41.143890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed001ff-f575-4271-b31c-e4a364a8bfa6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:41.170943 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-0ed001ff-f575-4271-b31c-e4a364a8bfa6 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:01:41.172442 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-0ed001ff-f575-4271-b31c-e4a364a8bfa6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:01:41.189605 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1243/2474] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:01:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:01:41.190801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5644f97-37f1-4875-ba5b-27fdd053d642 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:41.223967 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5644f97-37f1-4875-ba5b-27fdd053d642 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:41.243320 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583cc1b0-bff1-4adf-87f0-2daf9d685d99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:41.244521 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583cc1b0-bff1-4adf-87f0-2daf9d685d99 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:41.245532 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583cc1b0-bff1-4adf-87f0-2daf9d685d99 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:41.245993 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583cc1b0-bff1-4adf-87f0-2daf9d685d99 admin admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:41.247575 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-583cc1b0-bff1-4adf-87f0-2daf9d685d99 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 01:01:41.248304 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-583cc1b0-bff1-4adf-87f0-2daf9d685d99 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:41.251571 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-583cc1b0-bff1-4adf-87f0-2daf9d685d99 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:41.254847 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:01:41.286609 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1232/2475] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 01:01:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 01:01:41.287220 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed001ff-f575-4271-b31c-e4a364a8bfa6 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:41.302813 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed001ff-f575-4271-b31c-e4a364a8bfa6 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:41.312608 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd3a1c5-6414-4a4e-8920-549435f5555f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:41.314502 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd3a1c5-6414-4a4e-8920-549435f5555f admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:41.315237 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd3a1c5-6414-4a4e-8920-549435f5555f admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:41.316528 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd3a1c5-6414-4a4e-8920-549435f5555f admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:41.317196 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd3a1c5-6414-4a4e-8920-549435f5555f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:41.322181 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd3a1c5-6414-4a4e-8920-549435f5555f admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:41.382711 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1244/2476] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 01:01:41 2022] POST /identity/v3/projects => generated 371 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 01:01:41.387636 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583cc1b0-bff1-4adf-87f0-2daf9d685d99 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:41.408119 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583cc1b0-bff1-4adf-87f0-2daf9d685d99 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:41.478376 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c77b1a1f-840e-4c53-aa33-ffebe819e726 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:41.479114 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77b1a1f-840e-4c53-aa33-ffebe819e726 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:41.479462 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77b1a1f-840e-4c53-aa33-ffebe819e726 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:41.479801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77b1a1f-840e-4c53-aa33-ffebe819e726 admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:41.480579 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c77b1a1f-840e-4c53-aa33-ffebe819e726 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:41.482797 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c77b1a1f-840e-4c53-aa33-ffebe819e726 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:41.516372 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1233/2477] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 01:01:41 2022] POST /identity/v3/users => generated 286 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 01:01:41.520753 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd3a1c5-6414-4a4e-8920-549435f5555f admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:41.536130 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd3a1c5-6414-4a4e-8920-549435f5555f admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:41.544096 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627fe6a7-0506-4709-bf98-bec953167c14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:41.545159 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627fe6a7-0506-4709-bf98-bec953167c14 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:41.545690 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627fe6a7-0506-4709-bf98-bec953167c14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:41.546151 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627fe6a7-0506-4709-bf98-bec953167c14 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:41.547359 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-627fe6a7-0506-4709-bf98-bec953167c14 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:41.550545 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-627fe6a7-0506-4709-bf98-bec953167c14 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:41.572241 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1245/2478] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:01:41 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 01:01:41.576970 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c77b1a1f-840e-4c53-aa33-ffebe819e726 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:41.588611 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c77b1a1f-840e-4c53-aa33-ffebe819e726 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:41.593972 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9e7c72e-1608-45fb-8718-25f942e71898 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:41.594640 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e7c72e-1608-45fb-8718-25f942e71898 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:41.594935 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e7c72e-1608-45fb-8718-25f942e71898 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:41.595220 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e7c72e-1608-45fb-8718-25f942e71898 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:41.595952 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9e7c72e-1608-45fb-8718-25f942e71898 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:41.597648 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9e7c72e-1608-45fb-8718-25f942e71898 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:41.612204 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1234/2479] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:01:41 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 01:01:41.616515 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627fe6a7-0506-4709-bf98-bec953167c14 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:41.628203 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627fe6a7-0506-4709-bf98-bec953167c14 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:41.633104 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab4eb2c3-7697-4dc0-9cb9-5a721c15dfbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:41.633908 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4eb2c3-7697-4dc0-9cb9-5a721c15dfbe admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:41.634192 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4eb2c3-7697-4dc0-9cb9-5a721c15dfbe admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:41.634439 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4eb2c3-7697-4dc0-9cb9-5a721c15dfbe admin admin] PATH_INFO: `/v3/projects/69a3d6ce251e480d866a86ed9f6f2eb9/users/598ec83f7b7f4a9f90543218852aaf14/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:41.652284 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab4eb2c3-7697-4dc0-9cb9-5a721c15dfbe admin admin] RBAC: Authorizing `identity:create_grant(project_id=69a3d6ce251e480d866a86ed9f6f2eb9, user_id=598ec83f7b7f4a9f90543218852aaf14, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:41.655154 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab4eb2c3-7697-4dc0-9cb9-5a721c15dfbe admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:41.676464 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1246/2480] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 01:01:41 2022] PUT /identity/v3/projects/69a3d6ce251e480d866a86ed9f6f2eb9/users/598ec83f7b7f4a9f90543218852aaf14/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 01:01:41.681936 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ce2728-8cc3-482f-ac6f-d47651614e26 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:41.682376 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ce2728-8cc3-482f-ac6f-d47651614e26 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:41.682737 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ce2728-8cc3-482f-ac6f-d47651614e26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:41.705112 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-b0ce2728-8cc3-482f-ac6f-d47651614e26 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:01:41.706620 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-b0ce2728-8cc3-482f-ac6f-d47651614e26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:01:41.795945 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1235/2481] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:01:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:01:41.809188 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab4eb2c3-7697-4dc0-9cb9-5a721c15dfbe admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:41.812306 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-ab4eb2c3-7697-4dc0-9cb9-5a721c15dfbe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:01:41.846901 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:01:41.908437 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab4eb2c3-7697-4dc0-9cb9-5a721c15dfbe admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:41.913086 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d37526c-e091-4a45-bac0-c77addd5712f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:41.913533 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d37526c-e091-4a45-bac0-c77addd5712f service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:41.913816 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d37526c-e091-4a45-bac0-c77addd5712f service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:41.914090 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d37526c-e091-4a45-bac0-c77addd5712f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:41.925120 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d37526c-e091-4a45-bac0-c77addd5712f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:41.926616 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d37526c-e091-4a45-bac0-c77addd5712f service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:41.941285 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1247/2482] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:01:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:01:42.199389 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ce2728-8cc3-482f-ac6f-d47651614e26 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:42.202584 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-b0ce2728-8cc3-482f-ac6f-d47651614e26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:01:42.246547 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ce2728-8cc3-482f-ac6f-d47651614e26 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:42.270359 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc48915-4e42-42fd-95fb-b16af668901a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:42.282023 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc48915-4e42-42fd-95fb-b16af668901a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:42.282515 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc48915-4e42-42fd-95fb-b16af668901a admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:42.282985 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc48915-4e42-42fd-95fb-b16af668901a admin admin] PATH_INFO: `/v3/users/a642abaaaa8945d3a6b18f1c8eb08488` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:42.284820 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbc48915-4e42-42fd-95fb-b16af668901a admin admin] RBAC: Authorizing `identity:delete_user(user_id=a642abaaaa8945d3a6b18f1c8eb08488)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:42.288662 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbc48915-4e42-42fd-95fb-b16af668901a admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:42.412215 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-fbc48915-4e42-42fd-95fb-b16af668901a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a642abaaaa8945d3a6b18f1c8eb08488'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:42.423773 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-fbc48915-4e42-42fd-95fb-b16af668901a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a642abaaaa8945d3a6b18f1c8eb08488'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:42.424684 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-fbc48915-4e42-42fd-95fb-b16af668901a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a642abaaaa8945d3a6b18f1c8eb08488'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:42.457034 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-fbc48915-4e42-42fd-95fb-b16af668901a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a642abaaaa8945d3a6b18f1c8eb08488'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:42.461947 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-fbc48915-4e42-42fd-95fb-b16af668901a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a642abaaaa8945d3a6b18f1c8eb08488'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:42.472978 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=88,keystone:INSERT=4,keystone:DELETE=8 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:01:42.515785 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1236/2483] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 01:01:42 2022] DELETE /identity/v3/users/a642abaaaa8945d3a6b18f1c8eb08488 => generated 0 bytes in 318 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:01:42.739616 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d37526c-e091-4a45-bac0-c77addd5712f service neutron] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:42.742042 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-7d37526c-e091-4a45-bac0-c77addd5712f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:01:42.896419 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d37526c-e091-4a45-bac0-c77addd5712f service neutron] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:43.002209 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbde8039-bad7-4ad1-a0d9-e42715677ec9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f142988a4dd14a5f96eacea45c046f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:43.002856 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbde8039-bad7-4ad1-a0d9-e42715677ec9 service nova] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:43.003164 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbde8039-bad7-4ad1-a0d9-e42715677ec9 service nova] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:43.003507 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbde8039-bad7-4ad1-a0d9-e42715677ec9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:43.005550 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-bbde8039-bad7-4ad1-a0d9-e42715677ec9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:01:43.021899 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbde8039-bad7-4ad1-a0d9-e42715677ec9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:43.027021 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbde8039-bad7-4ad1-a0d9-e42715677ec9 service nova] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:43.049655 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1248/2484] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:01:42 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:01:43.208418 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc48915-4e42-42fd-95fb-b16af668901a admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:43.210252 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-fbc48915-4e42-42fd-95fb-b16af668901a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:01:43.351264 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc48915-4e42-42fd-95fb-b16af668901a admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:43.368322 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68775115-3e1c-4178-845b-d03fccfcdcd4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:43.369719 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68775115-3e1c-4178-845b-d03fccfcdcd4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:43.370209 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68775115-3e1c-4178-845b-d03fccfcdcd4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:43.370940 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68775115-3e1c-4178-845b-d03fccfcdcd4 admin admin] PATH_INFO: `/v3/projects/a0010b18914c438d9828a77f9fa2237e` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:43.378512 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68775115-3e1c-4178-845b-d03fccfcdcd4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a0010b18914c438d9828a77f9fa2237e)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:43.382435 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68775115-3e1c-4178-845b-d03fccfcdcd4 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:43.520332 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-68775115-3e1c-4178-845b-d03fccfcdcd4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a0010b18914c438d9828a77f9fa2237e', 'request_id': 'req-68775115-3e1c-4178-845b-d03fccfcdcd4'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:43.535201 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-68775115-3e1c-4178-845b-d03fccfcdcd4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a0010b18914c438d9828a77f9fa2237e', 'request_id': 'req-68775115-3e1c-4178-845b-d03fccfcdcd4'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:43.583341 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-68775115-3e1c-4178-845b-d03fccfcdcd4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a0010b18914c438d9828a77f9fa2237e', 'request_id': 'req-68775115-3e1c-4178-845b-d03fccfcdcd4'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:43.591745 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65e1f94-9057-4006-af94-38cf93d95bfe None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:43.591745 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65e1f94-9057-4006-af94-38cf93d95bfe None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:43.591745 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65e1f94-9057-4006-af94-38cf93d95bfe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:43.605280 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-68775115-3e1c-4178-845b-d03fccfcdcd4 admin admin] The token cache is being invalidate because project a0010b18914c438d9828a77f9fa2237e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81386) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 01:01:43.605835 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-68775115-3e1c-4178-845b-d03fccfcdcd4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:43.607847 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-68775115-3e1c-4178-845b-d03fccfcdcd4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:43.613744 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1237/2485] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 01:01:43 2022] DELETE /identity/v3/projects/a0010b18914c438d9828a77f9fa2237e => generated 0 bytes in 406 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:01:43.622465 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68775115-3e1c-4178-845b-d03fccfcdcd4 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:43.629202 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-68775115-3e1c-4178-845b-d03fccfcdcd4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:01:43.647048 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-c65e1f94-9057-4006-af94-38cf93d95bfe None None] MFA Rules not processed for user `3f20ffd8df994c1db1b1e7248db5b5a3`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:01:43.651273 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-c65e1f94-9057-4006-af94-38cf93d95bfe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:01:43.795135 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:01:43.826845 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1249/2486] 10.222.0.52 () {58 vars in 1009 bytes} [Sun Sep 25 01:01:43 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 241 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:01:43.849383 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c65e1f94-9057-4006-af94-38cf93d95bfe None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:43.854828 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-c65e1f94-9057-4006-af94-38cf93d95bfe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:01:43.864158 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68775115-3e1c-4178-845b-d03fccfcdcd4 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:43.880235 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93bc99e8-cbf8-4f23-ae0b-dcd1aa9d6e77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:43.881196 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93bc99e8-cbf8-4f23-ae0b-dcd1aa9d6e77 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:43.881833 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93bc99e8-cbf8-4f23-ae0b-dcd1aa9d6e77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:43.882421 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93bc99e8-cbf8-4f23-ae0b-dcd1aa9d6e77 admin admin] PATH_INFO: `/v3/users/29d87d3093d44b83850f8128f84f99cd` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:43.884774 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93bc99e8-cbf8-4f23-ae0b-dcd1aa9d6e77 admin admin] RBAC: Authorizing `identity:delete_user(user_id=29d87d3093d44b83850f8128f84f99cd)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:43.889116 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93bc99e8-cbf8-4f23-ae0b-dcd1aa9d6e77 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:43.932721 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:01:44.012353 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-93bc99e8-cbf8-4f23-ae0b-dcd1aa9d6e77 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '29d87d3093d44b83850f8128f84f99cd'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:44.023839 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-93bc99e8-cbf8-4f23-ae0b-dcd1aa9d6e77 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '29d87d3093d44b83850f8128f84f99cd'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:44.024921 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-93bc99e8-cbf8-4f23-ae0b-dcd1aa9d6e77 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '29d87d3093d44b83850f8128f84f99cd'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:44.047725 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-93bc99e8-cbf8-4f23-ae0b-dcd1aa9d6e77 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '29d87d3093d44b83850f8128f84f99cd'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:44.051075 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-93bc99e8-cbf8-4f23-ae0b-dcd1aa9d6e77 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '29d87d3093d44b83850f8128f84f99cd'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:44.109679 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c65e1f94-9057-4006-af94-38cf93d95bfe None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:44.135284 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a0bb05-650b-434b-a9f3-8d70b9799e90 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da606b3906dd4171bd76fd281e03e258', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:44.137540 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a0bb05-650b-434b-a9f3-8d70b9799e90 service swift] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:44.138708 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a0bb05-650b-434b-a9f3-8d70b9799e90 service swift] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:44.139650 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a0bb05-650b-434b-a9f3-8d70b9799e90 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:44.143066 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1238/2487] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 01:01:43 2022] DELETE /identity/v3/users/29d87d3093d44b83850f8128f84f99cd => generated 0 bytes in 511 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:01:44.152118 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-39a0bb05-650b-434b-a9f3-8d70b9799e90 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:01:44.179713 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39a0bb05-650b-434b-a9f3-8d70b9799e90 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:44.184114 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39a0bb05-650b-434b-a9f3-8d70b9799e90 service swift] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:44.212648 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1250/2488] 10.222.0.52 () {60 vars in 1384 bytes} [Sun Sep 25 01:01:43 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 365 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 25 01:01:44.604681 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93bc99e8-cbf8-4f23-ae0b-dcd1aa9d6e77 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:44.612554 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-93bc99e8-cbf8-4f23-ae0b-dcd1aa9d6e77 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:01:44.942273 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93bc99e8-cbf8-4f23-ae0b-dcd1aa9d6e77 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:44.958161 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb75dc4-5d84-422a-ac54-28dd0b193c77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:44.959353 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb75dc4-5d84-422a-ac54-28dd0b193c77 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:44.959758 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb75dc4-5d84-422a-ac54-28dd0b193c77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:44.960300 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb75dc4-5d84-422a-ac54-28dd0b193c77 admin admin] PATH_INFO: `/v3/projects/72ebfef0864443e598916e212e9811df` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:44.964293 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb75dc4-5d84-422a-ac54-28dd0b193c77 admin admin] RBAC: Authorizing `identity:delete_project(project_id=72ebfef0864443e598916e212e9811df)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:44.968216 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb75dc4-5d84-422a-ac54-28dd0b193c77 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:45.112089 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-6fb75dc4-5d84-422a-ac54-28dd0b193c77 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '72ebfef0864443e598916e212e9811df', 'request_id': 'req-6fb75dc4-5d84-422a-ac54-28dd0b193c77'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:45.130030 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-6fb75dc4-5d84-422a-ac54-28dd0b193c77 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '72ebfef0864443e598916e212e9811df', 'request_id': 'req-6fb75dc4-5d84-422a-ac54-28dd0b193c77'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:45.142964 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-6fb75dc4-5d84-422a-ac54-28dd0b193c77 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '72ebfef0864443e598916e212e9811df', 'request_id': 'req-6fb75dc4-5d84-422a-ac54-28dd0b193c77'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:45.163858 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-6fb75dc4-5d84-422a-ac54-28dd0b193c77 admin admin] The token cache is being invalidate because project 72ebfef0864443e598916e212e9811df was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81386) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 01:01:45.164208 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-6fb75dc4-5d84-422a-ac54-28dd0b193c77 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:45.165905 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-6fb75dc4-5d84-422a-ac54-28dd0b193c77 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:45.167969 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1239/2489] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 01:01:44 2022] DELETE /identity/v3/projects/72ebfef0864443e598916e212e9811df => generated 0 bytes in 564 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:01:45.175672 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b4d487-fde6-49f2-bc39-c1955fc525a3 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:45.176559 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b4d487-fde6-49f2-bc39-c1955fc525a3 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:45.176991 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b4d487-fde6-49f2-bc39-c1955fc525a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:45.218049 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-22b4d487-fde6-49f2-bc39-c1955fc525a3 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:01:45.219350 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-22b4d487-fde6-49f2-bc39-c1955fc525a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:01:45.356625 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1251/2490] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:01:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:01:45.371750 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695d01e1-c7b5-4605-af4c-0423cd1f11f5 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:45.372273 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695d01e1-c7b5-4605-af4c-0423cd1f11f5 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:45.372709 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695d01e1-c7b5-4605-af4c-0423cd1f11f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:45.400318 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-695d01e1-c7b5-4605-af4c-0423cd1f11f5 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:01:45.401346 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-695d01e1-c7b5-4605-af4c-0423cd1f11f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:01:45.418822 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1240/2491] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:01:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:01:45.424232 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b4d487-fde6-49f2-bc39-c1955fc525a3 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:45.439188 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b4d487-fde6-49f2-bc39-c1955fc525a3 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:45.448894 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44895be7-3b08-499d-8155-4a628ba8efeb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:45.448894 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44895be7-3b08-499d-8155-4a628ba8efeb admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:45.448894 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44895be7-3b08-499d-8155-4a628ba8efeb admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:45.448894 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44895be7-3b08-499d-8155-4a628ba8efeb admin admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:45.449676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44895be7-3b08-499d-8155-4a628ba8efeb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 01:01:45.450479 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44895be7-3b08-499d-8155-4a628ba8efeb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:45.452919 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44895be7-3b08-499d-8155-4a628ba8efeb admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:45.482762 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1252/2492] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 01:01:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 01:01:45.494589 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695d01e1-c7b5-4605-af4c-0423cd1f11f5 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:45.516198 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695d01e1-c7b5-4605-af4c-0423cd1f11f5 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:45.525741 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3eac9ca-3a3a-4c63-aa53-064efdcc4f0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:45.526809 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3eac9ca-3a3a-4c63-aa53-064efdcc4f0b admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:45.527229 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3eac9ca-3a3a-4c63-aa53-064efdcc4f0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:45.527689 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3eac9ca-3a3a-4c63-aa53-064efdcc4f0b admin admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:45.528863 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3eac9ca-3a3a-4c63-aa53-064efdcc4f0b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:45.532549 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3eac9ca-3a3a-4c63-aa53-064efdcc4f0b admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:45.567487 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:DELETE=16,keystone:SELECT=81,keystone:INSERT=3 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:01:45.630387 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44895be7-3b08-499d-8155-4a628ba8efeb admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:45.632882 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1241/2493] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 01:01:45 2022] POST /identity/v3/projects => generated 391 bytes in 145 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 01:01:45.644087 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44895be7-3b08-499d-8155-4a628ba8efeb admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:45.841001 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77956fdf-0e26-4842-9dca-a59ca7e7850f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:45.841995 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77956fdf-0e26-4842-9dca-a59ca7e7850f admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:45.842555 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77956fdf-0e26-4842-9dca-a59ca7e7850f admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:45.843046 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77956fdf-0e26-4842-9dca-a59ca7e7850f admin admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:45.844184 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77956fdf-0e26-4842-9dca-a59ca7e7850f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:45.847134 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77956fdf-0e26-4842-9dca-a59ca7e7850f admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:45.880679 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:01:45.904675 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1253/2494] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 01:01:45 2022] POST /identity/v3/users => generated 296 bytes in 275 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 01:01:45.912818 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3eac9ca-3a3a-4c63-aa53-064efdcc4f0b admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:45.937512 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3eac9ca-3a3a-4c63-aa53-064efdcc4f0b admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:45.945874 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe33203-ef94-41d4-a862-f37e7cfe2c1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:45.949615 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe33203-ef94-41d4-a862-f37e7cfe2c1c admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:45.950208 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe33203-ef94-41d4-a862-f37e7cfe2c1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:45.950779 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe33203-ef94-41d4-a862-f37e7cfe2c1c admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:45.960172 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe33203-ef94-41d4-a862-f37e7cfe2c1c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:45.969240 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe33203-ef94-41d4-a862-f37e7cfe2c1c admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:46.009534 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1242/2495] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:01:45 2022] GET /identity/v3/roles => generated 1420 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 01:01:46.017985 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77956fdf-0e26-4842-9dca-a59ca7e7850f admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:46.037163 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77956fdf-0e26-4842-9dca-a59ca7e7850f admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:46.049847 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-936c1250-fb44-4085-9236-a80af30a1351 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:46.050489 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936c1250-fb44-4085-9236-a80af30a1351 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:46.050787 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936c1250-fb44-4085-9236-a80af30a1351 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:46.051033 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936c1250-fb44-4085-9236-a80af30a1351 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:46.051883 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-936c1250-fb44-4085-9236-a80af30a1351 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:46.053617 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-936c1250-fb44-4085-9236-a80af30a1351 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:46.090366 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1254/2496] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:01:46 2022] GET /identity/v3/roles => generated 1420 bytes in 70 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 01:01:46.092889 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe33203-ef94-41d4-a862-f37e7cfe2c1c admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:46.162125 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe33203-ef94-41d4-a862-f37e7cfe2c1c admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:46.178718 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e6815d7-ad6c-4144-8ffd-a8c85bbcfa73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:46.179410 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6815d7-ad6c-4144-8ffd-a8c85bbcfa73 admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:46.179727 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6815d7-ad6c-4144-8ffd-a8c85bbcfa73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:46.179997 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6815d7-ad6c-4144-8ffd-a8c85bbcfa73 admin admin] PATH_INFO: `/v3/projects/c6554f9cd2bb4cdb96f76d1f87468962/users/94dca156406c42b9892a301d67fee90a/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:46.249559 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e6815d7-ad6c-4144-8ffd-a8c85bbcfa73 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c6554f9cd2bb4cdb96f76d1f87468962, user_id=94dca156406c42b9892a301d67fee90a, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:46.252784 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e6815d7-ad6c-4144-8ffd-a8c85bbcfa73 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:46.281701 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1243/2497] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 01:01:46 2022] PUT /identity/v3/projects/c6554f9cd2bb4cdb96f76d1f87468962/users/94dca156406c42b9892a301d67fee90a/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:01:46.306213 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe28290-edd4-4877-b835-ffcbc470d065 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:46.307149 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe28290-edd4-4877-b835-ffcbc470d065 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:46.307598 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe28290-edd4-4877-b835-ffcbc470d065 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:46.375865 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-7fe28290-edd4-4877-b835-ffcbc470d065 None None] MFA Rules not processed for user `94dca156406c42b9892a301d67fee90a`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:01:46.377423 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-7fe28290-edd4-4877-b835-ffcbc470d065 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:01:46.531371 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1255/2498] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:01:46 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 226 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:01:46.537583 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e6815d7-ad6c-4144-8ffd-a8c85bbcfa73 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:46.772204 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e6815d7-ad6c-4144-8ffd-a8c85bbcfa73 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:46.790206 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3b700a5-7825-40ae-9df2-f730732db8ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:46.790900 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b700a5-7825-40ae-9df2-f730732db8ae admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:46.791186 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b700a5-7825-40ae-9df2-f730732db8ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:46.791447 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b700a5-7825-40ae-9df2-f730732db8ae admin admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:46.792105 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3b700a5-7825-40ae-9df2-f730732db8ae admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:46.794161 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3b700a5-7825-40ae-9df2-f730732db8ae admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:46.871374 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1244/2499] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 01:01:46 2022] POST /identity/v3/projects => generated 393 bytes in 334 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 01:01:46.877610 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe28290-edd4-4877-b835-ffcbc470d065 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:46.902361 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe28290-edd4-4877-b835-ffcbc470d065 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:47.086745 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a630328-03e9-41a1-b7a9-2ce0ff4af2e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:47.087642 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a630328-03e9-41a1-b7a9-2ce0ff4af2e9 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:47.088135 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a630328-03e9-41a1-b7a9-2ce0ff4af2e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:47.088905 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a630328-03e9-41a1-b7a9-2ce0ff4af2e9 admin admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:47.090816 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a630328-03e9-41a1-b7a9-2ce0ff4af2e9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:47.095045 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a630328-03e9-41a1-b7a9-2ce0ff4af2e9 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:47.158227 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3b700a5-7825-40ae-9df2-f730732db8ae admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:47.159886 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1256/2500] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 01:01:46 2022] POST /identity/v3/users => generated 303 bytes in 275 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 01:01:47.209744 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3b700a5-7825-40ae-9df2-f730732db8ae admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:47.224163 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-619fcb0f-b214-402f-919c-70628cfe982a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:47.225547 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-619fcb0f-b214-402f-919c-70628cfe982a admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:47.226122 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-619fcb0f-b214-402f-919c-70628cfe982a admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:47.226669 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-619fcb0f-b214-402f-919c-70628cfe982a admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:47.228534 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-619fcb0f-b214-402f-919c-70628cfe982a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:47.232217 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-619fcb0f-b214-402f-919c-70628cfe982a admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:47.268355 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a630328-03e9-41a1-b7a9-2ce0ff4af2e9 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:47.279588 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1245/2501] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:01:47 2022] GET /identity/v3/roles => generated 1420 bytes in 120 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 01:01:47.289750 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a630328-03e9-41a1-b7a9-2ce0ff4af2e9 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:47.298804 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7495edce-ee26-4b19-ae1b-77305f3b9198 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:47.300110 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7495edce-ee26-4b19-ae1b-77305f3b9198 admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:47.300673 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7495edce-ee26-4b19-ae1b-77305f3b9198 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:47.301323 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7495edce-ee26-4b19-ae1b-77305f3b9198 admin admin] PATH_INFO: `/v3/projects/2ec79525454f495dadd6199cdcfdc8fb/users/7298cbc97c384a7b887433569a10b3a0/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:47.355355 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7495edce-ee26-4b19-ae1b-77305f3b9198 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2ec79525454f495dadd6199cdcfdc8fb, user_id=7298cbc97c384a7b887433569a10b3a0, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:47.358896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7495edce-ee26-4b19-ae1b-77305f3b9198 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:47.403759 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1257/2502] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 01:01:47 2022] PUT /identity/v3/projects/2ec79525454f495dadd6199cdcfdc8fb/users/7298cbc97c384a7b887433569a10b3a0/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:01:47.406788 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-619fcb0f-b214-402f-919c-70628cfe982a admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:47.426400 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-619fcb0f-b214-402f-919c-70628cfe982a admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:47.651317 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6a98c5-c166-4006-8834-a2a56831877f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:47.652573 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6a98c5-c166-4006-8834-a2a56831877f admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:47.653508 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6a98c5-c166-4006-8834-a2a56831877f admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:47.655509 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6a98c5-c166-4006-8834-a2a56831877f admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:47.657140 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa6a98c5-c166-4006-8834-a2a56831877f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:47.668854 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa6a98c5-c166-4006-8834-a2a56831877f admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:47.738416 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7495edce-ee26-4b19-ae1b-77305f3b9198 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:47.742347 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1246/2503] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:01:47 2022] GET /identity/v3/roles => generated 1420 bytes in 325 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 01:01:47.753742 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7495edce-ee26-4b19-ae1b-77305f3b9198 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:47.771904 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf5a7691-e7d9-4905-887b-306fde5c8178 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:47.772572 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5a7691-e7d9-4905-887b-306fde5c8178 admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:47.772943 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5a7691-e7d9-4905-887b-306fde5c8178 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:47.773298 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5a7691-e7d9-4905-887b-306fde5c8178 admin admin] PATH_INFO: `/v3/projects/2ec79525454f495dadd6199cdcfdc8fb/users/7298cbc97c384a7b887433569a10b3a0/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:47.780101 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf5a7691-e7d9-4905-887b-306fde5c8178 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2ec79525454f495dadd6199cdcfdc8fb, user_id=7298cbc97c384a7b887433569a10b3a0, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:47.783534 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf5a7691-e7d9-4905-887b-306fde5c8178 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:47.829348 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1258/2504] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 01:01:47 2022] PUT /identity/v3/projects/2ec79525454f495dadd6199cdcfdc8fb/users/7298cbc97c384a7b887433569a10b3a0/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 76 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:01:47.839770 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f6200d-5f26-4556-8e6e-5ba6bf1e744c None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:47.840846 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f6200d-5f26-4556-8e6e-5ba6bf1e744c None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:47.841505 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f6200d-5f26-4556-8e6e-5ba6bf1e744c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:47.923319 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-82f6200d-5f26-4556-8e6e-5ba6bf1e744c None None] MFA Rules not processed for user `7298cbc97c384a7b887433569a10b3a0`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:01:47.924978 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-82f6200d-5f26-4556-8e6e-5ba6bf1e744c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:01:47.977772 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:01:48.116347 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1247/2505] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:01:47 2022] POST /identity/v3/auth/tokens => generated 3337 bytes in 279 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:01:48.130715 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf5a7691-e7d9-4905-887b-306fde5c8178 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:48.133775 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-bf5a7691-e7d9-4905-887b-306fde5c8178 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:01:48.307114 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:01:48.400661 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf5a7691-e7d9-4905-887b-306fde5c8178 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:48.409501 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb741c78-de1a-4480-b2dd-1ab6b06ab739 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:48.410481 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb741c78-de1a-4480-b2dd-1ab6b06ab739 service cinder] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:48.410980 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb741c78-de1a-4480-b2dd-1ab6b06ab739 service cinder] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:48.411451 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb741c78-de1a-4480-b2dd-1ab6b06ab739 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:48.427163 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb741c78-de1a-4480-b2dd-1ab6b06ab739 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:48.429470 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb741c78-de1a-4480-b2dd-1ab6b06ab739 service cinder] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:48.450428 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1259/2506] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:01:48 2022] GET /identity/v3/auth/tokens => generated 3337 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 01:01:48.572095 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f6200d-5f26-4556-8e6e-5ba6bf1e744c None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:48.574032 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-82f6200d-5f26-4556-8e6e-5ba6bf1e744c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:01:48.605182 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f6200d-5f26-4556-8e6e-5ba6bf1e744c None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:48.609724 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce63617e-5159-4bf9-9c15-a87839519a30 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:48.610710 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce63617e-5159-4bf9-9c15-a87839519a30 service cinder] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:48.611505 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce63617e-5159-4bf9-9c15-a87839519a30 service cinder] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:48.611505 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce63617e-5159-4bf9-9c15-a87839519a30 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:48.833501 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce63617e-5159-4bf9-9c15-a87839519a30 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:48.845358 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce63617e-5159-4bf9-9c15-a87839519a30 service cinder] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:48.903454 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1248/2507] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:01:48 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 332 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 01:01:54.672298 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8a8652-8a31-43f3-aed9-e658bf362803 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:54.673304 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8a8652-8a31-43f3-aed9-e658bf362803 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:54.673762 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8a8652-8a31-43f3-aed9-e658bf362803 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:54.769007 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-0b8a8652-8a31-43f3-aed9-e658bf362803 None None] MFA Rules not processed for user `598ec83f7b7f4a9f90543218852aaf14`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:01:54.771369 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-0b8a8652-8a31-43f3-aed9-e658bf362803 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:01:55.137825 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1260/2508] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:01:54 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 466 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 01:01:55.146269 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce63617e-5159-4bf9-9c15-a87839519a30 service cinder] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:55.151753 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-ce63617e-5159-4bf9-9c15-a87839519a30 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:01:55.464832 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce63617e-5159-4bf9-9c15-a87839519a30 service cinder] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:55.475885 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a54c89-5131-4578-bfbb-463fb7f99c5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:55.477658 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a54c89-5131-4578-bfbb-463fb7f99c5b admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:55.486035 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a54c89-5131-4578-bfbb-463fb7f99c5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:55.486035 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a54c89-5131-4578-bfbb-463fb7f99c5b admin admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:55.486035 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a54c89-5131-4578-bfbb-463fb7f99c5b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:55.486035 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a54c89-5131-4578-bfbb-463fb7f99c5b admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:55.589183 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1249/2509] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 01:01:55 2022] POST /identity/v3/projects => generated 373 bytes in 446 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 01:01:55.591361 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b8a8652-8a31-43f3-aed9-e658bf362803 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:55.611404 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a54c89-5131-4578-bfbb-463fb7f99c5b admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:55.640335 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b8a8652-8a31-43f3-aed9-e658bf362803 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:55.839113 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:01:55.912328 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a54c89-5131-4578-bfbb-463fb7f99c5b admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:55.913103 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b27e0d05-c42d-4096-b49b-2e78b437448f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:55.927815 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083b2d7b-1ae3-4f28-ba88-bc9079d56970 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:55.931792 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083b2d7b-1ae3-4f28-ba88-bc9079d56970 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:55.932314 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083b2d7b-1ae3-4f28-ba88-bc9079d56970 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:55.932780 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083b2d7b-1ae3-4f28-ba88-bc9079d56970 admin admin] PATH_INFO: `/v3/users/94dca156406c42b9892a301d67fee90a` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:55.934749 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27e0d05-c42d-4096-b49b-2e78b437448f admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:55.935079 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27e0d05-c42d-4096-b49b-2e78b437448f admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:55.935296 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-083b2d7b-1ae3-4f28-ba88-bc9079d56970 admin admin] RBAC: Authorizing `identity:delete_user(user_id=94dca156406c42b9892a301d67fee90a)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:55.935482 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27e0d05-c42d-4096-b49b-2e78b437448f admin admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:55.939463 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-083b2d7b-1ae3-4f28-ba88-bc9079d56970 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:55.942244 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b27e0d05-c42d-4096-b49b-2e78b437448f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:55.950253 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b27e0d05-c42d-4096-b49b-2e78b437448f admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:55.998061 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1261/2510] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 01:01:55 2022] POST /identity/v3/users => generated 293 bytes in 408 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 01:01:56.005375 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b27e0d05-c42d-4096-b49b-2e78b437448f admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:56.036967 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b27e0d05-c42d-4096-b49b-2e78b437448f admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:56.048672 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45861aba-44e9-4a38-b36e-401b27868f11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:56.049681 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45861aba-44e9-4a38-b36e-401b27868f11 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:56.050233 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45861aba-44e9-4a38-b36e-401b27868f11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:56.050987 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45861aba-44e9-4a38-b36e-401b27868f11 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:56.052488 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45861aba-44e9-4a38-b36e-401b27868f11 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:56.057281 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45861aba-44e9-4a38-b36e-401b27868f11 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:56.081293 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-083b2d7b-1ae3-4f28-ba88-bc9079d56970 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '94dca156406c42b9892a301d67fee90a'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:56.089927 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1262/2511] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:01:56 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 01:01:56.105743 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45861aba-44e9-4a38-b36e-401b27868f11 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:56.131846 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-083b2d7b-1ae3-4f28-ba88-bc9079d56970 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '94dca156406c42b9892a301d67fee90a'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:56.135097 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-083b2d7b-1ae3-4f28-ba88-bc9079d56970 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '94dca156406c42b9892a301d67fee90a'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:56.170888 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-083b2d7b-1ae3-4f28-ba88-bc9079d56970 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '94dca156406c42b9892a301d67fee90a'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:56.174846 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-083b2d7b-1ae3-4f28-ba88-bc9079d56970 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '94dca156406c42b9892a301d67fee90a'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:56.189009 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45861aba-44e9-4a38-b36e-401b27868f11 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:56.209229 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de6628c-62e3-4102-8858-56ac2dcce510 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:56.210847 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de6628c-62e3-4102-8858-56ac2dcce510 admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:56.220899 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de6628c-62e3-4102-8858-56ac2dcce510 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:56.221835 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de6628c-62e3-4102-8858-56ac2dcce510 admin admin] PATH_INFO: `/v3/projects/2fcc4e24aaf24b7f8d61af98d0fd2431/users/de85ef7ef549463a88db6ff041f83574/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:56.240226 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1250/2512] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 01:01:55 2022] DELETE /identity/v3/users/94dca156406c42b9892a301d67fee90a => generated 0 bytes in 635 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:01:56.247257 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c849154-df65-4853-bf1c-8d3c155f53b0 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:56.247556 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c849154-df65-4853-bf1c-8d3c155f53b0 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:56.247850 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c849154-df65-4853-bf1c-8d3c155f53b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:56.313950 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-6c849154-df65-4853-bf1c-8d3c155f53b0 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:01:56.315177 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-6c849154-df65-4853-bf1c-8d3c155f53b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:01:56.359580 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de6628c-62e3-4102-8858-56ac2dcce510 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2fcc4e24aaf24b7f8d61af98d0fd2431, user_id=de85ef7ef549463a88db6ff041f83574, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:56.373291 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de6628c-62e3-4102-8858-56ac2dcce510 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:56.444879 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1263/2513] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 01:01:56 2022] PUT /identity/v3/projects/2fcc4e24aaf24b7f8d61af98d0fd2431/users/de85ef7ef549463a88db6ff041f83574/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 344 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:01:56.451498 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de6628c-62e3-4102-8858-56ac2dcce510 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:56.459563 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-9de6628c-62e3-4102-8858-56ac2dcce510 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:01:56.534340 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1251/2514] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:01:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 288 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:01:56.558911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:01:56.567526 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c849154-df65-4853-bf1c-8d3c155f53b0 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:56.570915 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-6c849154-df65-4853-bf1c-8d3c155f53b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:01:56.847740 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c849154-df65-4853-bf1c-8d3c155f53b0 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:56.890637 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:01:56.896418 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de6628c-62e3-4102-8858-56ac2dcce510 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:56.927668 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ba97230-139e-45e8-9d31-024d227301b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:56.928661 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba97230-139e-45e8-9d31-024d227301b7 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:56.929137 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba97230-139e-45e8-9d31-024d227301b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:56.930112 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba97230-139e-45e8-9d31-024d227301b7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:56.932392 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ba97230-139e-45e8-9d31-024d227301b7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:56.943107 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ba97230-139e-45e8-9d31-024d227301b7 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:56.971750 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1264/2515] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:01:56 2022] GET /identity/v3/roles => generated 1420 bytes in 521 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 01:01:56.978909 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ba97230-139e-45e8-9d31-024d227301b7 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:57.010862 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ba97230-139e-45e8-9d31-024d227301b7 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:57.023332 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a1c2a9-0ddc-4799-8717-2564f6a4e5f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:57.025962 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a1c2a9-0ddc-4799-8717-2564f6a4e5f9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:57.026673 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a1c2a9-0ddc-4799-8717-2564f6a4e5f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:57.027282 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a1c2a9-0ddc-4799-8717-2564f6a4e5f9 admin admin] PATH_INFO: `/v3/projects/2fcc4e24aaf24b7f8d61af98d0fd2431/users/de85ef7ef549463a88db6ff041f83574/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:57.032366 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20a1c2a9-0ddc-4799-8717-2564f6a4e5f9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2fcc4e24aaf24b7f8d61af98d0fd2431, user_id=de85ef7ef549463a88db6ff041f83574, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:57.043426 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20a1c2a9-0ddc-4799-8717-2564f6a4e5f9 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:57.068186 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc1a7c09-e006-43ca-a0f5-e05adeb5d2ce service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:57.070260 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1265/2516] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 01:01:56 2022] PUT /identity/v3/projects/2fcc4e24aaf24b7f8d61af98d0fd2431/users/de85ef7ef549463a88db6ff041f83574/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 92 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:01:57.078566 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1a7c09-e006-43ca-a0f5-e05adeb5d2ce service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:57.079023 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1a7c09-e006-43ca-a0f5-e05adeb5d2ce service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:57.084987 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1a7c09-e006-43ca-a0f5-e05adeb5d2ce service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:57.155839 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc1a7c09-e006-43ca-a0f5-e05adeb5d2ce service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:57.163169 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc1a7c09-e006-43ca-a0f5-e05adeb5d2ce service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:57.204861 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1252/2517] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:01:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 640 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 01:01:58.962606 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a1c2a9-0ddc-4799-8717-2564f6a4e5f9 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:58.966455 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-20a1c2a9-0ddc-4799-8717-2564f6a4e5f9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:01:59.305688 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a1c2a9-0ddc-4799-8717-2564f6a4e5f9 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:59.317397 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fcc1135-7125-4531-8a1a-7d66d2446e9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:59.318524 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fcc1135-7125-4531-8a1a-7d66d2446e9a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:59.319228 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fcc1135-7125-4531-8a1a-7d66d2446e9a admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:59.319886 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fcc1135-7125-4531-8a1a-7d66d2446e9a admin admin] PATH_INFO: `/v3/projects/c6554f9cd2bb4cdb96f76d1f87468962` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:59.322444 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fcc1135-7125-4531-8a1a-7d66d2446e9a admin admin] RBAC: Authorizing `identity:delete_project(project_id=c6554f9cd2bb4cdb96f76d1f87468962)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:59.325146 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fcc1135-7125-4531-8a1a-7d66d2446e9a admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:01:59.419119 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-0fcc1135-7125-4531-8a1a-7d66d2446e9a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c6554f9cd2bb4cdb96f76d1f87468962', 'request_id': 'req-0fcc1135-7125-4531-8a1a-7d66d2446e9a'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:59.444589 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-0fcc1135-7125-4531-8a1a-7d66d2446e9a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c6554f9cd2bb4cdb96f76d1f87468962', 'request_id': 'req-0fcc1135-7125-4531-8a1a-7d66d2446e9a'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:59.458890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-0fcc1135-7125-4531-8a1a-7d66d2446e9a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c6554f9cd2bb4cdb96f76d1f87468962', 'request_id': 'req-0fcc1135-7125-4531-8a1a-7d66d2446e9a'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:59.510403 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-0fcc1135-7125-4531-8a1a-7d66d2446e9a admin admin] The token cache is being invalidate because project c6554f9cd2bb4cdb96f76d1f87468962 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 01:01:59.514727 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-0fcc1135-7125-4531-8a1a-7d66d2446e9a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:59.518212 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-0fcc1135-7125-4531-8a1a-7d66d2446e9a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:01:59.526146 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1266/2518] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 01:01:58 2022] DELETE /identity/v3/projects/c6554f9cd2bb4cdb96f76d1f87468962 => generated 0 bytes in 564 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:01:59.530716 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc1a7c09-e006-43ca-a0f5-e05adeb5d2ce service neutron] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:01:59.591973 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc1a7c09-e006-43ca-a0f5-e05adeb5d2ce service neutron] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:01:59.862877 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b733db-398e-49c5-beb1-ca5435bdc255 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:01:59.864344 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b733db-398e-49c5-beb1-ca5435bdc255 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:01:59.865111 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b733db-398e-49c5-beb1-ca5435bdc255 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:01:59.865906 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b733db-398e-49c5-beb1-ca5435bdc255 admin admin] PATH_INFO: `/v3/users/7298cbc97c384a7b887433569a10b3a0` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:01:59.868592 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05b733db-398e-49c5-beb1-ca5435bdc255 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7298cbc97c384a7b887433569a10b3a0)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:01:59.873079 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05b733db-398e-49c5-beb1-ca5435bdc255 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:00.001887 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-05b733db-398e-49c5-beb1-ca5435bdc255 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7298cbc97c384a7b887433569a10b3a0'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:00.019602 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-05b733db-398e-49c5-beb1-ca5435bdc255 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7298cbc97c384a7b887433569a10b3a0'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:00.020480 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-05b733db-398e-49c5-beb1-ca5435bdc255 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7298cbc97c384a7b887433569a10b3a0'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:00.036135 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-05b733db-398e-49c5-beb1-ca5435bdc255 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7298cbc97c384a7b887433569a10b3a0'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:00.040122 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-05b733db-398e-49c5-beb1-ca5435bdc255 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7298cbc97c384a7b887433569a10b3a0'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:00.093847 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1253/2519] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 01:01:59 2022] DELETE /identity/v3/users/7298cbc97c384a7b887433569a10b3a0 => generated 0 bytes in 561 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:02:03.843676 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fcc1135-7125-4531-8a1a-7d66d2446e9a admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:03.845917 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-0fcc1135-7125-4531-8a1a-7d66d2446e9a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:02:03.882169 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fcc1135-7125-4531-8a1a-7d66d2446e9a admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:03.969915 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:02:04.075489 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c8be67-de6b-405b-b917-e1641a54c7a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:04.076595 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c8be67-de6b-405b-b917-e1641a54c7a3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:04.077508 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c8be67-de6b-405b-b917-e1641a54c7a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:04.078034 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c8be67-de6b-405b-b917-e1641a54c7a3 admin admin] PATH_INFO: `/v3/projects/2ec79525454f495dadd6199cdcfdc8fb` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:04.080121 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04c8be67-de6b-405b-b917-e1641a54c7a3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2ec79525454f495dadd6199cdcfdc8fb)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:04.084892 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04c8be67-de6b-405b-b917-e1641a54c7a3 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:04.318737 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-04c8be67-de6b-405b-b917-e1641a54c7a3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2ec79525454f495dadd6199cdcfdc8fb', 'request_id': 'req-04c8be67-de6b-405b-b917-e1641a54c7a3'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:04.345001 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-04c8be67-de6b-405b-b917-e1641a54c7a3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2ec79525454f495dadd6199cdcfdc8fb', 'request_id': 'req-04c8be67-de6b-405b-b917-e1641a54c7a3'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:04.383657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-04c8be67-de6b-405b-b917-e1641a54c7a3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2ec79525454f495dadd6199cdcfdc8fb', 'request_id': 'req-04c8be67-de6b-405b-b917-e1641a54c7a3'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:04.444484 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-04c8be67-de6b-405b-b917-e1641a54c7a3 admin admin] The token cache is being invalidate because project 2ec79525454f495dadd6199cdcfdc8fb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 01:02:04.449704 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-04c8be67-de6b-405b-b917-e1641a54c7a3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:04.453192 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-04c8be67-de6b-405b-b917-e1641a54c7a3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:04.468911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1267/2520] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 01:02:03 2022] DELETE /identity/v3/projects/2ec79525454f495dadd6199cdcfdc8fb => generated 0 bytes in 622 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:02:04.471165 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb014c7-8c1d-43bf-b58c-577130cdaae2 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:04.471707 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb014c7-8c1d-43bf-b58c-577130cdaae2 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:04.471707 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb014c7-8c1d-43bf-b58c-577130cdaae2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:04.559448 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-ceb014c7-8c1d-43bf-b58c-577130cdaae2 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:02:04.562725 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-ceb014c7-8c1d-43bf-b58c-577130cdaae2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:02:04.718286 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:02:04.847329 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5145268-0ef2-4cc4-b25c-aba58c97f024 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:04.848629 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5145268-0ef2-4cc4-b25c-aba58c97f024 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:04.849164 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5145268-0ef2-4cc4-b25c-aba58c97f024 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:04.854048 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1254/2521] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:02:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 384 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 01:02:04.898763 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-c5145268-0ef2-4cc4-b25c-aba58c97f024 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:02:04.902315 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-c5145268-0ef2-4cc4-b25c-aba58c97f024 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:02:04.936842 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1268/2522] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:02:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:02:04.940977 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb014c7-8c1d-43bf-b58c-577130cdaae2 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:04.955692 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb014c7-8c1d-43bf-b58c-577130cdaae2 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:04.962792 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08932ea2-87cb-4310-b047-437aa9b5ab13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:04.963701 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08932ea2-87cb-4310-b047-437aa9b5ab13 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:04.964294 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08932ea2-87cb-4310-b047-437aa9b5ab13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:04.964742 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08932ea2-87cb-4310-b047-437aa9b5ab13 admin admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:04.965610 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08932ea2-87cb-4310-b047-437aa9b5ab13 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 01:02:04.966096 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08932ea2-87cb-4310-b047-437aa9b5ab13 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:04.968150 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08932ea2-87cb-4310-b047-437aa9b5ab13 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:04.984337 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1255/2523] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 01:02:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 01:02:04.989394 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5145268-0ef2-4cc4-b25c-aba58c97f024 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:05.005229 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5145268-0ef2-4cc4-b25c-aba58c97f024 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:05.021204 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ab25b9e-972b-43dd-a40c-c29b27e92492 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:05.022210 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab25b9e-972b-43dd-a40c-c29b27e92492 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:05.022649 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab25b9e-972b-43dd-a40c-c29b27e92492 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:05.023161 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab25b9e-972b-43dd-a40c-c29b27e92492 admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:05.024051 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ab25b9e-972b-43dd-a40c-c29b27e92492 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:05.026320 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ab25b9e-972b-43dd-a40c-c29b27e92492 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:05.086030 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1269/2524] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 01:02:04 2022] POST /identity/v3/projects => generated 385 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 01:02:05.090602 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08932ea2-87cb-4310-b047-437aa9b5ab13 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:05.116159 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08932ea2-87cb-4310-b047-437aa9b5ab13 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:05.244788 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed41f18-7a69-4803-aa5f-e5bfc3931a1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:05.245826 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed41f18-7a69-4803-aa5f-e5bfc3931a1e admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:05.246265 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed41f18-7a69-4803-aa5f-e5bfc3931a1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:05.246717 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed41f18-7a69-4803-aa5f-e5bfc3931a1e admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:05.247721 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ed41f18-7a69-4803-aa5f-e5bfc3931a1e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:05.250133 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ed41f18-7a69-4803-aa5f-e5bfc3931a1e admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:05.293916 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1256/2525] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 01:02:05 2022] POST /identity/v3/users => generated 293 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 01:02:05.299906 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ab25b9e-972b-43dd-a40c-c29b27e92492 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:05.315644 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ab25b9e-972b-43dd-a40c-c29b27e92492 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:05.322847 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-409a11f3-c38b-435a-beca-151318976e8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:05.323763 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409a11f3-c38b-435a-beca-151318976e8f admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:05.324253 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409a11f3-c38b-435a-beca-151318976e8f admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:05.324680 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409a11f3-c38b-435a-beca-151318976e8f admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:05.325606 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-409a11f3-c38b-435a-beca-151318976e8f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:05.328383 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-409a11f3-c38b-435a-beca-151318976e8f admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:05.382664 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1270/2526] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:02:05 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 01:02:05.384779 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed41f18-7a69-4803-aa5f-e5bfc3931a1e admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:05.420824 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed41f18-7a69-4803-aa5f-e5bfc3931a1e admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:05.454714 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e0a1c5-a731-4812-b96b-db25fe51410e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:05.455893 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e0a1c5-a731-4812-b96b-db25fe51410e admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:05.456370 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e0a1c5-a731-4812-b96b-db25fe51410e admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:05.456844 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e0a1c5-a731-4812-b96b-db25fe51410e admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:05.461775 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e0a1c5-a731-4812-b96b-db25fe51410e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:05.464440 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e0a1c5-a731-4812-b96b-db25fe51410e admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:05.560896 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-409a11f3-c38b-435a-beca-151318976e8f admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:05.571951 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1257/2527] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:02:05 2022] GET /identity/v3/roles => generated 1420 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 01:02:05.576959 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-409a11f3-c38b-435a-beca-151318976e8f admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:05.595266 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-230a4c1c-0a6a-4ae4-9544-11a5694683ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:05.602531 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230a4c1c-0a6a-4ae4-9544-11a5694683ff admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:05.603046 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230a4c1c-0a6a-4ae4-9544-11a5694683ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:05.603430 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230a4c1c-0a6a-4ae4-9544-11a5694683ff admin admin] PATH_INFO: `/v3/projects/6855896c54924c2d82431b162b1f72de/users/ef62223c55a4459e95a20636d20ca0db/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:05.650634 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-230a4c1c-0a6a-4ae4-9544-11a5694683ff admin admin] RBAC: Authorizing `identity:create_grant(project_id=6855896c54924c2d82431b162b1f72de, user_id=ef62223c55a4459e95a20636d20ca0db, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:05.654485 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-230a4c1c-0a6a-4ae4-9544-11a5694683ff admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:05.701152 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f1f84e-13dc-4752-a8b9-10af9a99cdae None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:05.701611 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f1f84e-13dc-4752-a8b9-10af9a99cdae None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:05.701995 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f1f84e-13dc-4752-a8b9-10af9a99cdae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:05.705496 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1271/2528] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 01:02:05 2022] PUT /identity/v3/projects/6855896c54924c2d82431b162b1f72de/users/ef62223c55a4459e95a20636d20ca0db/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:02:05.772552 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-85f1f84e-13dc-4752-a8b9-10af9a99cdae None None] MFA Rules not processed for user `ef62223c55a4459e95a20636d20ca0db`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:02:05.774094 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-85f1f84e-13dc-4752-a8b9-10af9a99cdae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:02:05.998687 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-230a4c1c-0a6a-4ae4-9544-11a5694683ff admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:06.017808 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1258/2529] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:02:05 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 294 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:02:06.129185 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:02:06.153910 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-230a4c1c-0a6a-4ae4-9544-11a5694683ff admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:06.165205 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-147cb94a-7706-4670-8cf3-88c3d0a0e14c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:06.166185 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147cb94a-7706-4670-8cf3-88c3d0a0e14c admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:06.166434 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147cb94a-7706-4670-8cf3-88c3d0a0e14c admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:06.166848 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147cb94a-7706-4670-8cf3-88c3d0a0e14c admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:06.167662 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-147cb94a-7706-4670-8cf3-88c3d0a0e14c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:06.176162 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-147cb94a-7706-4670-8cf3-88c3d0a0e14c admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:06.252192 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1272/2530] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 01:02:05 2022] POST /identity/v3/projects => generated 385 bytes in 254 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 01:02:06.254048 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85f1f84e-13dc-4752-a8b9-10af9a99cdae None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:06.330209 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85f1f84e-13dc-4752-a8b9-10af9a99cdae None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:06.438373 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:02:06.539070 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ccd638-c0ce-4388-b007-214688b61d08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:06.541709 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ccd638-c0ce-4388-b007-214688b61d08 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:06.542346 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ccd638-c0ce-4388-b007-214688b61d08 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:06.542799 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ccd638-c0ce-4388-b007-214688b61d08 admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:06.543656 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1ccd638-c0ce-4388-b007-214688b61d08 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:06.547769 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1ccd638-c0ce-4388-b007-214688b61d08 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:06.591747 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1259/2531] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 01:02:06 2022] POST /identity/v3/users => generated 299 bytes in 339 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 01:02:06.596056 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-147cb94a-7706-4670-8cf3-88c3d0a0e14c admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:06.610552 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-147cb94a-7706-4670-8cf3-88c3d0a0e14c admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:06.617345 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa33f2cc-eeb8-43e8-8df4-6a81b171bb0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:06.618722 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa33f2cc-eeb8-43e8-8df4-6a81b171bb0d admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:06.618722 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa33f2cc-eeb8-43e8-8df4-6a81b171bb0d admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:06.618722 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa33f2cc-eeb8-43e8-8df4-6a81b171bb0d admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:06.619272 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa33f2cc-eeb8-43e8-8df4-6a81b171bb0d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:06.621165 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa33f2cc-eeb8-43e8-8df4-6a81b171bb0d admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:06.645948 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1273/2532] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:02:06 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 01:02:06.647452 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ccd638-c0ce-4388-b007-214688b61d08 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:06.667281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ccd638-c0ce-4388-b007-214688b61d08 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:06.676135 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-012841b0-df29-45cb-9370-8c6ad6be588c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:06.677120 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012841b0-df29-45cb-9370-8c6ad6be588c admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:06.677624 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012841b0-df29-45cb-9370-8c6ad6be588c admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:06.678306 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012841b0-df29-45cb-9370-8c6ad6be588c admin admin] PATH_INFO: `/v3/projects/5a586042098641a89cceffac62777bf5/users/23b4b1482adf4ff18cab378a4d5f9244/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:06.708346 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-012841b0-df29-45cb-9370-8c6ad6be588c admin admin] RBAC: Authorizing `identity:create_grant(project_id=5a586042098641a89cceffac62777bf5, user_id=23b4b1482adf4ff18cab378a4d5f9244, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:06.712226 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-012841b0-df29-45cb-9370-8c6ad6be588c admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:06.746918 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa33f2cc-eeb8-43e8-8df4-6a81b171bb0d admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:06.748515 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1260/2533] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 01:02:06 2022] PUT /identity/v3/projects/5a586042098641a89cceffac62777bf5/users/23b4b1482adf4ff18cab378a4d5f9244/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 95 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:02:06.763123 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa33f2cc-eeb8-43e8-8df4-6a81b171bb0d admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:06.936796 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-229f131b-bd88-4928-b1d7-687fb0ed7917 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:06.938254 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-229f131b-bd88-4928-b1d7-687fb0ed7917 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:06.938254 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-229f131b-bd88-4928-b1d7-687fb0ed7917 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:06.942140 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-229f131b-bd88-4928-b1d7-687fb0ed7917 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:06.946450 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-229f131b-bd88-4928-b1d7-687fb0ed7917 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:06.952751 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-229f131b-bd88-4928-b1d7-687fb0ed7917 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:06.980311 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-012841b0-df29-45cb-9370-8c6ad6be588c admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:06.987466 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1274/2534] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:02:06 2022] GET /identity/v3/roles => generated 1420 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 01:02:07.002052 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-012841b0-df29-45cb-9370-8c6ad6be588c admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:07.037208 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c75c94-94dd-47ec-9f4a-5f390691c856 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:07.039926 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c75c94-94dd-47ec-9f4a-5f390691c856 admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:07.040979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c75c94-94dd-47ec-9f4a-5f390691c856 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:07.060107 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c75c94-94dd-47ec-9f4a-5f390691c856 admin admin] PATH_INFO: `/v3/projects/5a586042098641a89cceffac62777bf5/users/23b4b1482adf4ff18cab378a4d5f9244/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:07.084775 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5c75c94-94dd-47ec-9f4a-5f390691c856 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5a586042098641a89cceffac62777bf5, user_id=23b4b1482adf4ff18cab378a4d5f9244, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:07.109457 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5c75c94-94dd-47ec-9f4a-5f390691c856 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:07.160082 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1261/2535] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 01:02:06 2022] PUT /identity/v3/projects/5a586042098641a89cceffac62777bf5/users/23b4b1482adf4ff18cab378a4d5f9244/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:02:07.163135 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b48b23e-81c3-42cf-bb10-688da4a541c3 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:07.164847 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b48b23e-81c3-42cf-bb10-688da4a541c3 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:07.165380 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b48b23e-81c3-42cf-bb10-688da4a541c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:07.230053 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-9b48b23e-81c3-42cf-bb10-688da4a541c3 None None] MFA Rules not processed for user `23b4b1482adf4ff18cab378a4d5f9244`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:02:07.233534 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-9b48b23e-81c3-42cf-bb10-688da4a541c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:02:07.404950 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1275/2536] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:02:07 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 243 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:02:07.411936 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c75c94-94dd-47ec-9f4a-5f390691c856 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:07.415290 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-f5c75c94-94dd-47ec-9f4a-5f390691c856 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:02:07.683527 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c75c94-94dd-47ec-9f4a-5f390691c856 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:07.698863 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdbd0e69-e296-4b84-a6e4-20ac68eaaa7c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:07.699807 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdbd0e69-e296-4b84-a6e4-20ac68eaaa7c service cinder] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:07.700246 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdbd0e69-e296-4b84-a6e4-20ac68eaaa7c service cinder] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:07.700635 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdbd0e69-e296-4b84-a6e4-20ac68eaaa7c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:07.734686 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdbd0e69-e296-4b84-a6e4-20ac68eaaa7c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:07.737186 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdbd0e69-e296-4b84-a6e4-20ac68eaaa7c service cinder] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:07.772553 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1262/2537] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:02:07 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 361 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 01:02:09.271278 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b48b23e-81c3-42cf-bb10-688da4a541c3 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:09.461454 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:02:09.546959 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b48b23e-81c3-42cf-bb10-688da4a541c3 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:09.556454 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2906c380-2b2b-49af-a042-cfe6c050950f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:09.557541 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2906c380-2b2b-49af-a042-cfe6c050950f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:09.557975 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2906c380-2b2b-49af-a042-cfe6c050950f admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:09.559043 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2906c380-2b2b-49af-a042-cfe6c050950f admin admin] PATH_INFO: `/v3/users/ef62223c55a4459e95a20636d20ca0db` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:09.560632 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2906c380-2b2b-49af-a042-cfe6c050950f admin admin] RBAC: Authorizing `identity:delete_user(user_id=ef62223c55a4459e95a20636d20ca0db)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:09.563044 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2906c380-2b2b-49af-a042-cfe6c050950f admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:09.649323 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-2906c380-2b2b-49af-a042-cfe6c050950f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ef62223c55a4459e95a20636d20ca0db'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:09.650338 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-2906c380-2b2b-49af-a042-cfe6c050950f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ef62223c55a4459e95a20636d20ca0db'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:09.680040 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-2906c380-2b2b-49af-a042-cfe6c050950f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ef62223c55a4459e95a20636d20ca0db'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:09.681400 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-2906c380-2b2b-49af-a042-cfe6c050950f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ef62223c55a4459e95a20636d20ca0db'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:09.692608 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-2906c380-2b2b-49af-a042-cfe6c050950f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ef62223c55a4459e95a20636d20ca0db'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:09.738190 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1276/2538] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 01:02:09 2022] DELETE /identity/v3/users/ef62223c55a4459e95a20636d20ca0db => generated 0 bytes in 468 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:02:09.744872 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbcf975-1edb-46f0-bc55-9668d6aa7b3c None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:09.746161 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbcf975-1edb-46f0-bc55-9668d6aa7b3c None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:09.746738 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbcf975-1edb-46f0-bc55-9668d6aa7b3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:09.809605 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-5dbcf975-1edb-46f0-bc55-9668d6aa7b3c None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:02:09.812308 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-5dbcf975-1edb-46f0-bc55-9668d6aa7b3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:02:09.992989 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1263/2539] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:02:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 251 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:02:10.013587 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2906c380-2b2b-49af-a042-cfe6c050950f admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:10.015534 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-2906c380-2b2b-49af-a042-cfe6c050950f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:02:10.225484 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2906c380-2b2b-49af-a042-cfe6c050950f admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:10.235880 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ed76e45-f79d-434a-bded-aab538d20a49 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:10.236528 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed76e45-f79d-434a-bded-aab538d20a49 service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:10.236804 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed76e45-f79d-434a-bded-aab538d20a49 service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:10.237072 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed76e45-f79d-434a-bded-aab538d20a49 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:10.264480 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ed76e45-f79d-434a-bded-aab538d20a49 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:10.266706 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ed76e45-f79d-434a-bded-aab538d20a49 service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:10.296700 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1277/2540] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:02:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 284 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:02:11.159396 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dbcf975-1edb-46f0-bc55-9668d6aa7b3c None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:11.161762 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-5dbcf975-1edb-46f0-bc55-9668d6aa7b3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:02:11.214872 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dbcf975-1edb-46f0-bc55-9668d6aa7b3c None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:11.231438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c21e281-a2fa-4ae9-b835-7fbc9a3ced58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:11.232169 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c21e281-a2fa-4ae9-b835-7fbc9a3ced58 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:11.232610 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c21e281-a2fa-4ae9-b835-7fbc9a3ced58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:11.233363 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c21e281-a2fa-4ae9-b835-7fbc9a3ced58 admin admin] PATH_INFO: `/v3/projects/6855896c54924c2d82431b162b1f72de` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:11.238693 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c21e281-a2fa-4ae9-b835-7fbc9a3ced58 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6855896c54924c2d82431b162b1f72de)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:11.244440 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c21e281-a2fa-4ae9-b835-7fbc9a3ced58 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:11.384851 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=94,keystone:INSERT=5,keystone:DELETE=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:02:11.491426 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-1c21e281-a2fa-4ae9-b835-7fbc9a3ced58 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6855896c54924c2d82431b162b1f72de', 'request_id': 'req-1c21e281-a2fa-4ae9-b835-7fbc9a3ced58'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:11.526861 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-1c21e281-a2fa-4ae9-b835-7fbc9a3ced58 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6855896c54924c2d82431b162b1f72de', 'request_id': 'req-1c21e281-a2fa-4ae9-b835-7fbc9a3ced58'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:11.538904 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-1c21e281-a2fa-4ae9-b835-7fbc9a3ced58 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6855896c54924c2d82431b162b1f72de', 'request_id': 'req-1c21e281-a2fa-4ae9-b835-7fbc9a3ced58'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:11.589734 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-1c21e281-a2fa-4ae9-b835-7fbc9a3ced58 admin admin] The token cache is being invalidate because project 6855896c54924c2d82431b162b1f72de was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81386) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 01:02:11.590667 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-1c21e281-a2fa-4ae9-b835-7fbc9a3ced58 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:11.594196 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-1c21e281-a2fa-4ae9-b835-7fbc9a3ced58 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:11.599238 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1264/2541] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 01:02:11 2022] DELETE /identity/v3/projects/6855896c54924c2d82431b162b1f72de => generated 0 bytes in 441 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:02:11.604577 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ed76e45-f79d-434a-bded-aab538d20a49 service neutron] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:11.626035 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ed76e45-f79d-434a-bded-aab538d20a49 service neutron] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:11.754925 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4b07a5-d2b9-425e-b529-3d01e1937500 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:11.755489 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4b07a5-d2b9-425e-b529-3d01e1937500 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:11.755946 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4b07a5-d2b9-425e-b529-3d01e1937500 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:11.798290 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c25f593-7743-4d00-a6a8-95bd7e32e556 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:11.799439 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c25f593-7743-4d00-a6a8-95bd7e32e556 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:11.799900 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c25f593-7743-4d00-a6a8-95bd7e32e556 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:11.802045 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c25f593-7743-4d00-a6a8-95bd7e32e556 admin admin] PATH_INFO: `/v3/users/23b4b1482adf4ff18cab378a4d5f9244` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:11.803586 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c25f593-7743-4d00-a6a8-95bd7e32e556 admin admin] RBAC: Authorizing `identity:delete_user(user_id=23b4b1482adf4ff18cab378a4d5f9244)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:11.805877 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c25f593-7743-4d00-a6a8-95bd7e32e556 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:11.825236 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-cd4b07a5-d2b9-425e-b529-3d01e1937500 None None] MFA Rules not processed for user `de85ef7ef549463a88db6ff041f83574`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:02:11.827493 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:02:11.828335 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-cd4b07a5-d2b9-425e-b529-3d01e1937500 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:02:11.937873 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-1c25f593-7743-4d00-a6a8-95bd7e32e556 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '23b4b1482adf4ff18cab378a4d5f9244'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:11.939251 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-1c25f593-7743-4d00-a6a8-95bd7e32e556 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '23b4b1482adf4ff18cab378a4d5f9244'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:11.952524 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-1c25f593-7743-4d00-a6a8-95bd7e32e556 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '23b4b1482adf4ff18cab378a4d5f9244'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:11.955214 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-1c25f593-7743-4d00-a6a8-95bd7e32e556 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '23b4b1482adf4ff18cab378a4d5f9244'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:11.978992 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-1c25f593-7743-4d00-a6a8-95bd7e32e556 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '23b4b1482adf4ff18cab378a4d5f9244'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:12.027841 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1278/2542] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 01:02:11 2022] DELETE /identity/v3/users/23b4b1482adf4ff18cab378a4d5f9244 => generated 0 bytes in 424 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:02:12.194321 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1265/2543] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:02:11 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 435 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 01:02:12.207334 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c25f593-7743-4d00-a6a8-95bd7e32e556 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:12.207334 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-1c25f593-7743-4d00-a6a8-95bd7e32e556 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:02:12.433775 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c25f593-7743-4d00-a6a8-95bd7e32e556 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:12.453694 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1669b077-eefc-4985-bba0-b968a6f61726 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f142988a4dd14a5f96eacea45c046f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:12.454334 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1669b077-eefc-4985-bba0-b968a6f61726 service nova] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:12.454606 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1669b077-eefc-4985-bba0-b968a6f61726 service nova] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:12.454873 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1669b077-eefc-4985-bba0-b968a6f61726 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:12.457010 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-1669b077-eefc-4985-bba0-b968a6f61726 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:02:12.571124 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd4b07a5-d2b9-425e-b529-3d01e1937500 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:12.573448 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-cd4b07a5-d2b9-425e-b529-3d01e1937500 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:02:12.625610 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd4b07a5-d2b9-425e-b529-3d01e1937500 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:12.635385 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1669b077-eefc-4985-bba0-b968a6f61726 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:12.637228 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1669b077-eefc-4985-bba0-b968a6f61726 service nova] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:12.669046 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1279/2544] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:02:12 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 465 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:02:12.897622 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ee81f0-83b2-4c32-9c7c-f8afe73fd59e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:12.899591 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ee81f0-83b2-4c32-9c7c-f8afe73fd59e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:12.899591 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ee81f0-83b2-4c32-9c7c-f8afe73fd59e admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:12.900158 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ee81f0-83b2-4c32-9c7c-f8afe73fd59e admin admin] PATH_INFO: `/v3/projects/5a586042098641a89cceffac62777bf5` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:12.901741 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42ee81f0-83b2-4c32-9c7c-f8afe73fd59e admin admin] RBAC: Authorizing `identity:delete_project(project_id=5a586042098641a89cceffac62777bf5)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:12.910288 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42ee81f0-83b2-4c32-9c7c-f8afe73fd59e admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:13.027850 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:02:13.050418 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-42ee81f0-83b2-4c32-9c7c-f8afe73fd59e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5a586042098641a89cceffac62777bf5', 'request_id': 'req-42ee81f0-83b2-4c32-9c7c-f8afe73fd59e'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:13.069902 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-42ee81f0-83b2-4c32-9c7c-f8afe73fd59e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5a586042098641a89cceffac62777bf5', 'request_id': 'req-42ee81f0-83b2-4c32-9c7c-f8afe73fd59e'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:13.082989 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-42ee81f0-83b2-4c32-9c7c-f8afe73fd59e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5a586042098641a89cceffac62777bf5', 'request_id': 'req-42ee81f0-83b2-4c32-9c7c-f8afe73fd59e'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:13.115571 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-42ee81f0-83b2-4c32-9c7c-f8afe73fd59e admin admin] The token cache is being invalidate because project 5a586042098641a89cceffac62777bf5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81386) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 01:02:13.116033 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-42ee81f0-83b2-4c32-9c7c-f8afe73fd59e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:13.118310 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-42ee81f0-83b2-4c32-9c7c-f8afe73fd59e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:13.132834 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5c2660-67e0-4a68-870e-28429079880d None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:13.133379 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5c2660-67e0-4a68-870e-28429079880d None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:13.134369 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5c2660-67e0-4a68-870e-28429079880d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:13.141025 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1266/2545] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 01:02:12 2022] DELETE /identity/v3/projects/5a586042098641a89cceffac62777bf5 => generated 0 bytes in 556 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:02:13.166591 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-9a5c2660-67e0-4a68-870e-28429079880d None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:02:13.168304 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-9a5c2660-67e0-4a68-870e-28429079880d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:02:13.239983 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:02:13.383453 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1280/2546] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:02:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 253 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:02:13.392677 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2283f4c4-6187-4578-8516-e404a9bfed71 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:13.393616 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2283f4c4-6187-4578-8516-e404a9bfed71 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:13.394057 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2283f4c4-6187-4578-8516-e404a9bfed71 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:13.436869 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-2283f4c4-6187-4578-8516-e404a9bfed71 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:02:13.438657 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-2283f4c4-6187-4578-8516-e404a9bfed71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:02:13.467349 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1267/2547] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:02:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:02:13.468763 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5c2660-67e0-4a68-870e-28429079880d None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:13.524926 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5c2660-67e0-4a68-870e-28429079880d None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:13.548486 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac12136-2385-4b19-b8e8-b57afc94cb1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:13.549564 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac12136-2385-4b19-b8e8-b57afc94cb1c admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:13.550037 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac12136-2385-4b19-b8e8-b57afc94cb1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:13.550493 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac12136-2385-4b19-b8e8-b57afc94cb1c admin admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:13.551681 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bac12136-2385-4b19-b8e8-b57afc94cb1c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 01:02:13.552222 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bac12136-2385-4b19-b8e8-b57afc94cb1c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:13.555828 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bac12136-2385-4b19-b8e8-b57afc94cb1c admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:13.614499 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1281/2548] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 01:02:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 01:02:13.615138 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2283f4c4-6187-4578-8516-e404a9bfed71 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:13.629792 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2283f4c4-6187-4578-8516-e404a9bfed71 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:13.652715 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83abec1-008e-4ad7-a972-9347ea590edf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:13.658202 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83abec1-008e-4ad7-a972-9347ea590edf admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:13.658926 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83abec1-008e-4ad7-a972-9347ea590edf admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:13.659439 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83abec1-008e-4ad7-a972-9347ea590edf admin admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:13.666122 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a83abec1-008e-4ad7-a972-9347ea590edf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:13.668656 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a83abec1-008e-4ad7-a972-9347ea590edf admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:13.828722 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1268/2549] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 01:02:13 2022] POST /identity/v3/projects => generated 385 bytes in 214 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 01:02:13.834507 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac12136-2385-4b19-b8e8-b57afc94cb1c admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:13.859863 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac12136-2385-4b19-b8e8-b57afc94cb1c admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:14.039775 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09afe2b7-0ee1-40f4-9a0c-01697e538149 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:14.040899 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09afe2b7-0ee1-40f4-9a0c-01697e538149 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:14.041183 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09afe2b7-0ee1-40f4-9a0c-01697e538149 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:14.041495 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09afe2b7-0ee1-40f4-9a0c-01697e538149 admin admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:14.042283 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09afe2b7-0ee1-40f4-9a0c-01697e538149 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:14.048266 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09afe2b7-0ee1-40f4-9a0c-01697e538149 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:14.090196 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1282/2550] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 01:02:13 2022] POST /identity/v3/users => generated 293 bytes in 258 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 01:02:14.094926 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83abec1-008e-4ad7-a972-9347ea590edf admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:14.122794 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83abec1-008e-4ad7-a972-9347ea590edf admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:14.133091 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c73f23f-57bb-46fa-889f-4b2ddb38cdb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:14.134592 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c73f23f-57bb-46fa-889f-4b2ddb38cdb4 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:14.135145 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c73f23f-57bb-46fa-889f-4b2ddb38cdb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:14.135719 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c73f23f-57bb-46fa-889f-4b2ddb38cdb4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:14.137129 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c73f23f-57bb-46fa-889f-4b2ddb38cdb4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:14.140801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c73f23f-57bb-46fa-889f-4b2ddb38cdb4 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:14.172220 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1269/2551] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:02:14 2022] GET /identity/v3/roles => generated 1420 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 01:02:14.176804 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09afe2b7-0ee1-40f4-9a0c-01697e538149 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:14.209011 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09afe2b7-0ee1-40f4-9a0c-01697e538149 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:14.221370 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd09c951-e0cf-4fb5-a329-72985acdce33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:14.222612 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd09c951-e0cf-4fb5-a329-72985acdce33 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:14.223240 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd09c951-e0cf-4fb5-a329-72985acdce33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:14.223851 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd09c951-e0cf-4fb5-a329-72985acdce33 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:14.225223 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd09c951-e0cf-4fb5-a329-72985acdce33 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:14.231573 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd09c951-e0cf-4fb5-a329-72985acdce33 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:14.258951 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1283/2552] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:02:14 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 01:02:14.263506 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c73f23f-57bb-46fa-889f-4b2ddb38cdb4 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:14.310813 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c73f23f-57bb-46fa-889f-4b2ddb38cdb4 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:14.331746 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c99b8b5-e5ef-4d33-bf00-001663cb21c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:14.332399 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c99b8b5-e5ef-4d33-bf00-001663cb21c0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:14.332684 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c99b8b5-e5ef-4d33-bf00-001663cb21c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:14.333029 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c99b8b5-e5ef-4d33-bf00-001663cb21c0 admin admin] PATH_INFO: `/v3/projects/21d8d89a1819407984ff0b3b4d02fec1/users/c0b5c848beb04ba1b394e59dfb78d0bc/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:14.359738 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c99b8b5-e5ef-4d33-bf00-001663cb21c0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=21d8d89a1819407984ff0b3b4d02fec1, user_id=c0b5c848beb04ba1b394e59dfb78d0bc, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:14.362826 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c99b8b5-e5ef-4d33-bf00-001663cb21c0 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:14.406156 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1270/2553] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 01:02:14 2022] PUT /identity/v3/projects/21d8d89a1819407984ff0b3b4d02fec1/users/c0b5c848beb04ba1b394e59dfb78d0bc/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:02:14.409329 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f55e85-2918-42f1-afc0-1b9c72e99ed7 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:14.410150 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f55e85-2918-42f1-afc0-1b9c72e99ed7 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:14.410642 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f55e85-2918-42f1-afc0-1b9c72e99ed7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:14.474393 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-c0f55e85-2918-42f1-afc0-1b9c72e99ed7 None None] MFA Rules not processed for user `c0b5c848beb04ba1b394e59dfb78d0bc`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:02:14.476378 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-c0f55e85-2918-42f1-afc0-1b9c72e99ed7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:02:14.598510 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1284/2554] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:02:14 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:02:14.605725 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c99b8b5-e5ef-4d33-bf00-001663cb21c0 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:14.807271 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c99b8b5-e5ef-4d33-bf00-001663cb21c0 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:14.813829 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b24460f-1458-4531-8e51-5a2d25b11ac3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:14.814838 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b24460f-1458-4531-8e51-5a2d25b11ac3 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:14.815263 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b24460f-1458-4531-8e51-5a2d25b11ac3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:14.815640 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b24460f-1458-4531-8e51-5a2d25b11ac3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:14.816412 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b24460f-1458-4531-8e51-5a2d25b11ac3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:14.820294 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b24460f-1458-4531-8e51-5a2d25b11ac3 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:14.873337 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1271/2555] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 01:02:14 2022] POST /identity/v3/projects => generated 385 bytes in 265 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 01:02:14.875465 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f55e85-2918-42f1-afc0-1b9c72e99ed7 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:14.898486 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f55e85-2918-42f1-afc0-1b9c72e99ed7 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:14.959951 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:02:15.049833 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a9bcb6-6c85-4b51-9317-b2e1c7832c95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:15.050903 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a9bcb6-6c85-4b51-9317-b2e1c7832c95 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:15.051401 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a9bcb6-6c85-4b51-9317-b2e1c7832c95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:15.052451 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a9bcb6-6c85-4b51-9317-b2e1c7832c95 admin admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:15.053044 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a9bcb6-6c85-4b51-9317-b2e1c7832c95 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:15.057471 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a9bcb6-6c85-4b51-9317-b2e1c7832c95 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:15.097316 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1285/2556] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 01:02:14 2022] POST /identity/v3/users => generated 299 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 01:02:15.102843 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b24460f-1458-4531-8e51-5a2d25b11ac3 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:15.115782 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b24460f-1458-4531-8e51-5a2d25b11ac3 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:15.122214 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f1d7ceb-c491-4198-8cb6-f69c60127b70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:15.122851 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1d7ceb-c491-4198-8cb6-f69c60127b70 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:15.123120 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1d7ceb-c491-4198-8cb6-f69c60127b70 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:15.123396 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1d7ceb-c491-4198-8cb6-f69c60127b70 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:15.124214 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f1d7ceb-c491-4198-8cb6-f69c60127b70 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:15.125966 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f1d7ceb-c491-4198-8cb6-f69c60127b70 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:15.139093 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1272/2557] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:02:15 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 01:02:15.145509 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a9bcb6-6c85-4b51-9317-b2e1c7832c95 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:15.168494 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a9bcb6-6c85-4b51-9317-b2e1c7832c95 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:15.176640 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70100294-02dc-42eb-a240-0ca4b6da4a1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:15.177837 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70100294-02dc-42eb-a240-0ca4b6da4a1f admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:15.178317 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70100294-02dc-42eb-a240-0ca4b6da4a1f admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:15.180009 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70100294-02dc-42eb-a240-0ca4b6da4a1f admin admin] PATH_INFO: `/v3/projects/f5f655cf4e484f9583745ccc3a63ded5/users/ef63803f883841c9a1d25ff2b85a0400/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:15.213027 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70100294-02dc-42eb-a240-0ca4b6da4a1f admin admin] RBAC: Authorizing `identity:create_grant(project_id=f5f655cf4e484f9583745ccc3a63ded5, user_id=ef63803f883841c9a1d25ff2b85a0400, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:15.219488 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70100294-02dc-42eb-a240-0ca4b6da4a1f admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:15.256642 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1286/2558] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 01:02:15 2022] PUT /identity/v3/projects/f5f655cf4e484f9583745ccc3a63ded5/users/ef63803f883841c9a1d25ff2b85a0400/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:02:15.262689 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f1d7ceb-c491-4198-8cb6-f69c60127b70 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:15.280173 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f1d7ceb-c491-4198-8cb6-f69c60127b70 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:15.311643 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69fec9f-8eac-4419-96d1-13f7c80bc1bc None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:15.312094 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69fec9f-8eac-4419-96d1-13f7c80bc1bc None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:15.312537 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69fec9f-8eac-4419-96d1-13f7c80bc1bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:15.352209 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-c69fec9f-8eac-4419-96d1-13f7c80bc1bc None None] MFA Rules not processed for user `598ec83f7b7f4a9f90543218852aaf14`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:02:15.354171 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-c69fec9f-8eac-4419-96d1-13f7c80bc1bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:02:15.365840 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=92,keystone:DELETE=4,keystone:INSERT=4 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:02:15.436640 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-783490ed-384c-4dda-9ac3-08d3e1e75e68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:15.438801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783490ed-384c-4dda-9ac3-08d3e1e75e68 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:15.439500 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783490ed-384c-4dda-9ac3-08d3e1e75e68 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:15.440228 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783490ed-384c-4dda-9ac3-08d3e1e75e68 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:15.441775 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-783490ed-384c-4dda-9ac3-08d3e1e75e68 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:15.448148 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-783490ed-384c-4dda-9ac3-08d3e1e75e68 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:15.495288 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1273/2559] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:02:15 2022] GET /identity/v3/roles => generated 1420 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 01:02:15.498658 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-783490ed-384c-4dda-9ac3-08d3e1e75e68 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:15.519574 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1287/2560] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:02:15 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 01:02:15.530563 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-783490ed-384c-4dda-9ac3-08d3e1e75e68 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:15.536377 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4ef1de-20b1-43e8-baeb-5ee406c9d4fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:15.537798 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4ef1de-20b1-43e8-baeb-5ee406c9d4fd admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:15.538082 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4ef1de-20b1-43e8-baeb-5ee406c9d4fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:15.538890 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4ef1de-20b1-43e8-baeb-5ee406c9d4fd admin admin] PATH_INFO: `/v3/projects/f5f655cf4e484f9583745ccc3a63ded5/users/ef63803f883841c9a1d25ff2b85a0400/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:15.542493 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e4ef1de-20b1-43e8-baeb-5ee406c9d4fd admin admin] RBAC: Authorizing `identity:create_grant(project_id=f5f655cf4e484f9583745ccc3a63ded5, user_id=ef63803f883841c9a1d25ff2b85a0400, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:15.545209 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e4ef1de-20b1-43e8-baeb-5ee406c9d4fd admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:15.562533 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1274/2561] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 01:02:15 2022] PUT /identity/v3/projects/f5f655cf4e484f9583745ccc3a63ded5/users/ef63803f883841c9a1d25ff2b85a0400/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:02:15.575093 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5f1f67-dfa4-4254-b5a2-11d52e9c2964 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:15.575398 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5f1f67-dfa4-4254-b5a2-11d52e9c2964 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:15.575654 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5f1f67-dfa4-4254-b5a2-11d52e9c2964 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:15.620763 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-7e5f1f67-dfa4-4254-b5a2-11d52e9c2964 None None] MFA Rules not processed for user `ef63803f883841c9a1d25ff2b85a0400`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:02:15.622578 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-7e5f1f67-dfa4-4254-b5a2-11d52e9c2964 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:02:15.770770 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1288/2562] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:02:15 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:02:15.788416 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4ef1de-20b1-43e8-baeb-5ee406c9d4fd admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:15.791661 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-5e4ef1de-20b1-43e8-baeb-5ee406c9d4fd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:02:15.896579 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4ef1de-20b1-43e8-baeb-5ee406c9d4fd admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:15.903253 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d45dd0-e810-4d41-b9a4-e7d1d1b16796 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:15.904291 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d45dd0-e810-4d41-b9a4-e7d1d1b16796 service cinder] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:15.904741 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d45dd0-e810-4d41-b9a4-e7d1d1b16796 service cinder] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:15.905179 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d45dd0-e810-4d41-b9a4-e7d1d1b16796 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:16.003789 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03d45dd0-e810-4d41-b9a4-e7d1d1b16796 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:16.006757 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03d45dd0-e810-4d41-b9a4-e7d1d1b16796 service cinder] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:16.024506 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1275/2563] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:02:15 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 237 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:02:20.157938 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81681239-324b-42af-92a2-4bd8ac152302 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:20.159736 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81681239-324b-42af-92a2-4bd8ac152302 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:20.160459 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81681239-324b-42af-92a2-4bd8ac152302 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:20.163909 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1289/2564] 10.222.0.52 () {54 vars in 921 bytes} [Sun Sep 25 01:02:20 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 01:02:20.171357 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6acab470-ed35-4be7-ad6a-bd29174e7f3e None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:20.172956 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6acab470-ed35-4be7-ad6a-bd29174e7f3e None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:20.180961 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6acab470-ed35-4be7-ad6a-bd29174e7f3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:20.280272 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-6acab470-ed35-4be7-ad6a-bd29174e7f3e None None] MFA Rules not processed for user `f142988a4dd14a5f96eacea45c046f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:02:20.282598 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-6acab470-ed35-4be7-ad6a-bd29174e7f3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:02:20.449107 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1276/2565] 10.222.0.52 () {58 vars in 1018 bytes} [Sun Sep 25 01:02:20 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 280 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:02:20.466142 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81681239-324b-42af-92a2-4bd8ac152302 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:20.470140 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-81681239-324b-42af-92a2-4bd8ac152302 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:02:20.508984 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:02:20.557125 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81681239-324b-42af-92a2-4bd8ac152302 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:20.561075 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d46f2d22-d786-4987-97c2-6b318dde82dd service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4ebfc0abf1b842459a44ca9324252688', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:20.561726 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46f2d22-d786-4987-97c2-6b318dde82dd service glance] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:20.561996 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46f2d22-d786-4987-97c2-6b318dde82dd service glance] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:20.562263 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46f2d22-d786-4987-97c2-6b318dde82dd service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:20.574276 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d46f2d22-d786-4987-97c2-6b318dde82dd service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:20.575722 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d46f2d22-d786-4987-97c2-6b318dde82dd service glance] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:20.588306 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1290/2566] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:02:20 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:02:20.669147 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63d0395-121b-41ac-b9ff-5bda5f7020ab None None] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:20.669648 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63d0395-121b-41ac-b9ff-5bda5f7020ab None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:20.670124 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63d0395-121b-41ac-b9ff-5bda5f7020ab None None] PATH_INFO: `/` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:20.672006 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1277/2567] 10.222.0.52 () {54 vars in 921 bytes} [Sun Sep 25 01:02:20 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 01:02:20.677715 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d60bb1f6-0b00-4bd5-9c24-3019f7f7740b None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:20.678042 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d60bb1f6-0b00-4bd5-9c24-3019f7f7740b None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:20.678235 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d60bb1f6-0b00-4bd5-9c24-3019f7f7740b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:20.700226 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-d60bb1f6-0b00-4bd5-9c24-3019f7f7740b None None] MFA Rules not processed for user `20c274e96dec478b98ef51f80ea893b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:02:20.701729 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-d60bb1f6-0b00-4bd5-9c24-3019f7f7740b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:02:20.780156 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1291/2568] 10.222.0.52 () {58 vars in 1018 bytes} [Sun Sep 25 01:02:20 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:02:20.800294 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f63d0395-121b-41ac-b9ff-5bda5f7020ab None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:20.802540 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-f63d0395-121b-41ac-b9ff-5bda5f7020ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:02:20.819773 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f63d0395-121b-41ac-b9ff-5bda5f7020ab None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:20.824494 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e2c82eb-6068-45af-9c2c-3b843d5ec146 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:20.825403 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2c82eb-6068-45af-9c2c-3b843d5ec146 service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:20.825725 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2c82eb-6068-45af-9c2c-3b843d5ec146 service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:20.826062 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2c82eb-6068-45af-9c2c-3b843d5ec146 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:20.831047 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:02:20.835851 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e2c82eb-6068-45af-9c2c-3b843d5ec146 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:20.837480 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e2c82eb-6068-45af-9c2c-3b843d5ec146 service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:20.848476 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1278/2569] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:02:20 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:02:20.951760 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd49ac6-7cea-4771-a16c-a2390fa5a0c5 None None] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:20.952252 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd49ac6-7cea-4771-a16c-a2390fa5a0c5 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:20.952824 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd49ac6-7cea-4771-a16c-a2390fa5a0c5 None None] PATH_INFO: `/` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:20.955846 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1292/2570] 10.222.0.52 () {54 vars in 921 bytes} [Sun Sep 25 01:02:20 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 01:02:20.964157 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25f697f-09f4-43d8-a4fb-e50f76e96ba0 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:20.964614 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25f697f-09f4-43d8-a4fb-e50f76e96ba0 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:20.965008 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25f697f-09f4-43d8-a4fb-e50f76e96ba0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:20.997934 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-c25f697f-09f4-43d8-a4fb-e50f76e96ba0 None None] MFA Rules not processed for user `f142988a4dd14a5f96eacea45c046f85`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:02:20.999591 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-c25f697f-09f4-43d8-a4fb-e50f76e96ba0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:02:21.009427 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1279/2571] 10.222.0.52 () {58 vars in 1018 bytes} [Sun Sep 25 01:02:20 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:02:21.021861 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd49ac6-7cea-4771-a16c-a2390fa5a0c5 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:21.037606 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd49ac6-7cea-4771-a16c-a2390fa5a0c5 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:21.042094 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d01ba3-3cdc-415e-88b4-fb5f09331d47 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:21.042831 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d01ba3-3cdc-415e-88b4-fb5f09331d47 service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:21.043213 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d01ba3-3cdc-415e-88b4-fb5f09331d47 service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:21.043649 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d01ba3-3cdc-415e-88b4-fb5f09331d47 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:21.056901 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7d01ba3-3cdc-415e-88b4-fb5f09331d47 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:21.058413 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7d01ba3-3cdc-415e-88b4-fb5f09331d47 service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:21.071812 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1293/2572] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:02:21 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 01:02:21.492023 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25f697f-09f4-43d8-a4fb-e50f76e96ba0 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:21.493830 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-c25f697f-09f4-43d8-a4fb-e50f76e96ba0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:02:21.508673 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25f697f-09f4-43d8-a4fb-e50f76e96ba0 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:21.513353 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e149a0-7021-4071-a207-fb84d6a9df06 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:21.514114 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e149a0-7021-4071-a207-fb84d6a9df06 service cinder] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:21.515273 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e149a0-7021-4071-a207-fb84d6a9df06 service cinder] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:21.515634 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e149a0-7021-4071-a207-fb84d6a9df06 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:21.533171 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74e149a0-7021-4071-a207-fb84d6a9df06 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:21.534875 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74e149a0-7021-4071-a207-fb84d6a9df06 service cinder] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:21.557348 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1280/2573] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:02:21 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 59 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:02:23.804414 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd4b3af-7b54-4041-b6e7-755d64883038 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:23.805017 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd4b3af-7b54-4041-b6e7-755d64883038 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:23.805017 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd4b3af-7b54-4041-b6e7-755d64883038 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:23.827236 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-2bd4b3af-7b54-4041-b6e7-755d64883038 None None] MFA Rules not processed for user `3f20ffd8df994c1db1b1e7248db5b5a3`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:02:23.828536 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-2bd4b3af-7b54-4041-b6e7-755d64883038 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:02:23.929306 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1294/2574] 10.222.0.52 () {58 vars in 1009 bytes} [Sun Sep 25 01:02:23 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:02:23.964855 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e149a0-7021-4071-a207-fb84d6a9df06 service cinder] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:23.966510 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-74e149a0-7021-4071-a207-fb84d6a9df06 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:02:24.078953 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e149a0-7021-4071-a207-fb84d6a9df06 service cinder] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:24.089982 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361a45bc-5cd7-493b-a168-f8796690e945 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da606b3906dd4171bd76fd281e03e258', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:24.091073 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361a45bc-5cd7-493b-a168-f8796690e945 service swift] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:24.091242 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361a45bc-5cd7-493b-a168-f8796690e945 service swift] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:24.091658 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361a45bc-5cd7-493b-a168-f8796690e945 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:24.105903 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361a45bc-5cd7-493b-a168-f8796690e945 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:24.107825 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361a45bc-5cd7-493b-a168-f8796690e945 service swift] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:24.118796 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1281/2575] 10.222.0.52 () {60 vars in 1384 bytes} [Sun Sep 25 01:02:23 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 155 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 25 01:02:27.088046 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd4b3af-7b54-4041-b6e7-755d64883038 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:27.208378 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd4b3af-7b54-4041-b6e7-755d64883038 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:27.214506 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca559ce8-85ad-4678-bf7a-7bd76cb7c71b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:27.215164 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca559ce8-85ad-4678-bf7a-7bd76cb7c71b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:27.215445 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca559ce8-85ad-4678-bf7a-7bd76cb7c71b admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:27.215714 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca559ce8-85ad-4678-bf7a-7bd76cb7c71b admin admin] PATH_INFO: `/v3/users/c0b5c848beb04ba1b394e59dfb78d0bc` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:27.216986 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca559ce8-85ad-4678-bf7a-7bd76cb7c71b admin admin] RBAC: Authorizing `identity:delete_user(user_id=c0b5c848beb04ba1b394e59dfb78d0bc)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:27.218994 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca559ce8-85ad-4678-bf7a-7bd76cb7c71b admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:27.222511 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:02:27.299248 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-ca559ce8-85ad-4678-bf7a-7bd76cb7c71b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c0b5c848beb04ba1b394e59dfb78d0bc'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:27.300417 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-ca559ce8-85ad-4678-bf7a-7bd76cb7c71b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c0b5c848beb04ba1b394e59dfb78d0bc'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:27.314574 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-ca559ce8-85ad-4678-bf7a-7bd76cb7c71b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c0b5c848beb04ba1b394e59dfb78d0bc'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:27.315290 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-ca559ce8-85ad-4678-bf7a-7bd76cb7c71b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c0b5c848beb04ba1b394e59dfb78d0bc'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:27.326037 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-ca559ce8-85ad-4678-bf7a-7bd76cb7c71b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c0b5c848beb04ba1b394e59dfb78d0bc'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:27.364884 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1295/2576] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 01:02:27 2022] DELETE /identity/v3/users/c0b5c848beb04ba1b394e59dfb78d0bc => generated 0 bytes in 278 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:02:27.371318 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a186cdbf-b150-428d-ba67-dbf5f8dffa5a None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:27.372664 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a186cdbf-b150-428d-ba67-dbf5f8dffa5a None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:27.373402 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a186cdbf-b150-428d-ba67-dbf5f8dffa5a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:27.432079 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-a186cdbf-b150-428d-ba67-dbf5f8dffa5a None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:02:27.434660 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-a186cdbf-b150-428d-ba67-dbf5f8dffa5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:02:27.606403 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1282/2577] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:02:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 233 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:02:27.618126 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca559ce8-85ad-4678-bf7a-7bd76cb7c71b admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:27.620661 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-ca559ce8-85ad-4678-bf7a-7bd76cb7c71b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:02:27.746666 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca559ce8-85ad-4678-bf7a-7bd76cb7c71b admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:27.758818 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac7dd79f-f4e3-45d0-8553-545a538afbe1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:27.760596 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7dd79f-f4e3-45d0-8553-545a538afbe1 service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:27.761649 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7dd79f-f4e3-45d0-8553-545a538afbe1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:27.762423 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7dd79f-f4e3-45d0-8553-545a538afbe1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:27.784401 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac7dd79f-f4e3-45d0-8553-545a538afbe1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:27.786525 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac7dd79f-f4e3-45d0-8553-545a538afbe1 service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:27.799344 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1296/2578] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:02:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:02:28.544130 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a186cdbf-b150-428d-ba67-dbf5f8dffa5a None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:28.546567 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-a186cdbf-b150-428d-ba67-dbf5f8dffa5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:02:28.581236 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a186cdbf-b150-428d-ba67-dbf5f8dffa5a None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:28.610328 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b26d27-9a79-43f9-b763-ce168c13827b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:28.610997 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b26d27-9a79-43f9-b763-ce168c13827b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:28.611379 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b26d27-9a79-43f9-b763-ce168c13827b admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:28.611816 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b26d27-9a79-43f9-b763-ce168c13827b admin admin] PATH_INFO: `/v3/projects/21d8d89a1819407984ff0b3b4d02fec1` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:28.613491 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7b26d27-9a79-43f9-b763-ce168c13827b admin admin] RBAC: Authorizing `identity:delete_project(project_id=21d8d89a1819407984ff0b3b4d02fec1)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:28.616626 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7b26d27-9a79-43f9-b763-ce168c13827b admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:28.762288 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-b7b26d27-9a79-43f9-b763-ce168c13827b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '21d8d89a1819407984ff0b3b4d02fec1', 'request_id': 'req-b7b26d27-9a79-43f9-b763-ce168c13827b'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:28.785295 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-b7b26d27-9a79-43f9-b763-ce168c13827b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '21d8d89a1819407984ff0b3b4d02fec1', 'request_id': 'req-b7b26d27-9a79-43f9-b763-ce168c13827b'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:28.801004 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=93,keystone:DELETE=6,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:02:28.816619 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-b7b26d27-9a79-43f9-b763-ce168c13827b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '21d8d89a1819407984ff0b3b4d02fec1', 'request_id': 'req-b7b26d27-9a79-43f9-b763-ce168c13827b'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:28.842458 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-b7b26d27-9a79-43f9-b763-ce168c13827b admin admin] The token cache is being invalidate because project 21d8d89a1819407984ff0b3b4d02fec1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81386) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 01:02:28.842950 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-b7b26d27-9a79-43f9-b763-ce168c13827b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:28.844436 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-b7b26d27-9a79-43f9-b763-ce168c13827b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:28.852141 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1283/2579] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 01:02:28 2022] DELETE /identity/v3/projects/21d8d89a1819407984ff0b3b4d02fec1 => generated 0 bytes in 304 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:02:28.853503 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac7dd79f-f4e3-45d0-8553-545a538afbe1 service neutron] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:28.869495 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac7dd79f-f4e3-45d0-8553-545a538afbe1 service neutron] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:28.986270 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4bbb2fb-c92c-4db9-8b28-7df46510b309 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:28.987929 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4bbb2fb-c92c-4db9-8b28-7df46510b309 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:28.988353 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4bbb2fb-c92c-4db9-8b28-7df46510b309 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:28.988721 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4bbb2fb-c92c-4db9-8b28-7df46510b309 admin admin] PATH_INFO: `/v3/users/ef63803f883841c9a1d25ff2b85a0400` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:28.990648 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4bbb2fb-c92c-4db9-8b28-7df46510b309 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ef63803f883841c9a1d25ff2b85a0400)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:28.993029 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4bbb2fb-c92c-4db9-8b28-7df46510b309 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:29.034121 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:02:29.069906 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-a4bbb2fb-c92c-4db9-8b28-7df46510b309 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ef63803f883841c9a1d25ff2b85a0400'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:29.070968 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-a4bbb2fb-c92c-4db9-8b28-7df46510b309 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ef63803f883841c9a1d25ff2b85a0400'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:29.084419 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-a4bbb2fb-c92c-4db9-8b28-7df46510b309 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ef63803f883841c9a1d25ff2b85a0400'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:29.085308 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-a4bbb2fb-c92c-4db9-8b28-7df46510b309 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ef63803f883841c9a1d25ff2b85a0400'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:29.095334 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-a4bbb2fb-c92c-4db9-8b28-7df46510b309 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ef63803f883841c9a1d25ff2b85a0400'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:29.125041 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1297/2580] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 01:02:28 2022] DELETE /identity/v3/users/ef63803f883841c9a1d25ff2b85a0400 => generated 0 bytes in 272 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:02:29.639464 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b26d27-9a79-43f9-b763-ce168c13827b admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:29.657051 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-b7b26d27-9a79-43f9-b763-ce168c13827b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:02:29.689418 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b26d27-9a79-43f9-b763-ce168c13827b admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:29.890949 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a70e6a-65b3-46f3-8b97-15908371e4c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:29.892611 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a70e6a-65b3-46f3-8b97-15908371e4c3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:29.893221 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a70e6a-65b3-46f3-8b97-15908371e4c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:29.893974 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a70e6a-65b3-46f3-8b97-15908371e4c3 admin admin] PATH_INFO: `/v3/projects/f5f655cf4e484f9583745ccc3a63ded5` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:29.898805 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a70e6a-65b3-46f3-8b97-15908371e4c3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f5f655cf4e484f9583745ccc3a63ded5)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:29.901622 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a70e6a-65b3-46f3-8b97-15908371e4c3 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:30.039790 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-90a70e6a-65b3-46f3-8b97-15908371e4c3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f5f655cf4e484f9583745ccc3a63ded5', 'request_id': 'req-90a70e6a-65b3-46f3-8b97-15908371e4c3'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:30.057861 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-90a70e6a-65b3-46f3-8b97-15908371e4c3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f5f655cf4e484f9583745ccc3a63ded5', 'request_id': 'req-90a70e6a-65b3-46f3-8b97-15908371e4c3'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:30.099248 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-90a70e6a-65b3-46f3-8b97-15908371e4c3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f5f655cf4e484f9583745ccc3a63ded5', 'request_id': 'req-90a70e6a-65b3-46f3-8b97-15908371e4c3'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:30.142659 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-90a70e6a-65b3-46f3-8b97-15908371e4c3 admin admin] The token cache is being invalidate because project f5f655cf4e484f9583745ccc3a63ded5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81386) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 01:02:30.143066 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-90a70e6a-65b3-46f3-8b97-15908371e4c3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:30.144169 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-90a70e6a-65b3-46f3-8b97-15908371e4c3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:30.157654 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1284/2581] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 01:02:29 2022] DELETE /identity/v3/projects/f5f655cf4e484f9583745ccc3a63ded5 => generated 0 bytes in 512 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:02:30.161882 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42f8d91-c804-4144-a005-051f35381554 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:30.163115 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42f8d91-c804-4144-a005-051f35381554 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:30.163314 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42f8d91-c804-4144-a005-051f35381554 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:30.222861 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-b42f8d91-c804-4144-a005-051f35381554 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:02:30.226918 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-b42f8d91-c804-4144-a005-051f35381554 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:02:30.437442 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1298/2582] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:02:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 274 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:02:30.444810 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7acd92aa-97ed-476f-935a-f40c6394da7d None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:30.445216 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7acd92aa-97ed-476f-935a-f40c6394da7d None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:30.445696 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7acd92aa-97ed-476f-935a-f40c6394da7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:30.512584 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-7acd92aa-97ed-476f-935a-f40c6394da7d None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:02:30.515095 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-7acd92aa-97ed-476f-935a-f40c6394da7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:02:30.529393 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1285/2583] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:02:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 01:02:30.534081 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b42f8d91-c804-4144-a005-051f35381554 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:30.551471 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b42f8d91-c804-4144-a005-051f35381554 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:30.558407 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c3a3a99-c90f-45b5-a755-95c83f88b95b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:30.559334 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c3a3a99-c90f-45b5-a755-95c83f88b95b admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:30.559748 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c3a3a99-c90f-45b5-a755-95c83f88b95b admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:30.560163 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c3a3a99-c90f-45b5-a755-95c83f88b95b admin admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:30.561057 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c3a3a99-c90f-45b5-a755-95c83f88b95b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 01:02:30.561773 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c3a3a99-c90f-45b5-a755-95c83f88b95b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:30.563980 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c3a3a99-c90f-45b5-a755-95c83f88b95b admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:30.589567 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1299/2584] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 01:02:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 01:02:30.594731 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7acd92aa-97ed-476f-935a-f40c6394da7d None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:30.625616 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7acd92aa-97ed-476f-935a-f40c6394da7d None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:30.636253 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738ca4a0-244b-4c46-b783-277fe25c5db2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:30.638798 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738ca4a0-244b-4c46-b783-277fe25c5db2 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:30.639507 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738ca4a0-244b-4c46-b783-277fe25c5db2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:30.640177 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738ca4a0-244b-4c46-b783-277fe25c5db2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:30.641149 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-738ca4a0-244b-4c46-b783-277fe25c5db2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:30.645307 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-738ca4a0-244b-4c46-b783-277fe25c5db2 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:30.693378 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1286/2585] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 01:02:30 2022] POST /identity/v3/projects => generated 375 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 01:02:30.698473 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c3a3a99-c90f-45b5-a755-95c83f88b95b admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:30.712984 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c3a3a99-c90f-45b5-a755-95c83f88b95b admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:30.836223 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a13541f-b6e4-4cdf-ad5c-0de955a6fe5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:30.837239 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a13541f-b6e4-4cdf-ad5c-0de955a6fe5d admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:30.837949 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a13541f-b6e4-4cdf-ad5c-0de955a6fe5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:30.838335 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a13541f-b6e4-4cdf-ad5c-0de955a6fe5d admin admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:30.839397 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a13541f-b6e4-4cdf-ad5c-0de955a6fe5d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:30.848844 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a13541f-b6e4-4cdf-ad5c-0de955a6fe5d admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:30.903788 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1300/2586] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 01:02:30 2022] POST /identity/v3/users => generated 288 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 01:02:30.904923 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738ca4a0-244b-4c46-b783-277fe25c5db2 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:30.935252 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738ca4a0-244b-4c46-b783-277fe25c5db2 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:30.944433 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c659ed-7e80-4d4c-b228-b111b97a8115 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:30.945640 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c659ed-7e80-4d4c-b228-b111b97a8115 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:30.946207 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c659ed-7e80-4d4c-b228-b111b97a8115 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:30.946979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c659ed-7e80-4d4c-b228-b111b97a8115 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:30.948347 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21c659ed-7e80-4d4c-b228-b111b97a8115 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:30.952742 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21c659ed-7e80-4d4c-b228-b111b97a8115 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:30.983532 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1287/2587] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:02:30 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 01:02:30.990229 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a13541f-b6e4-4cdf-ad5c-0de955a6fe5d admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:31.004003 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a13541f-b6e4-4cdf-ad5c-0de955a6fe5d admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:31.022653 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79bb71dc-eca2-4666-8137-24b591e1422e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:31.024372 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79bb71dc-eca2-4666-8137-24b591e1422e admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:31.024849 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79bb71dc-eca2-4666-8137-24b591e1422e admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:31.025330 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79bb71dc-eca2-4666-8137-24b591e1422e admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:31.026536 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79bb71dc-eca2-4666-8137-24b591e1422e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:31.029743 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79bb71dc-eca2-4666-8137-24b591e1422e admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:31.061777 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1301/2588] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:02:30 2022] GET /identity/v3/roles => generated 1420 bytes in 73 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 01:02:31.084853 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c659ed-7e80-4d4c-b228-b111b97a8115 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:31.097359 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c659ed-7e80-4d4c-b228-b111b97a8115 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:31.114980 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e83e8800-42f1-4f32-8b28-18dd2b81a68e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:31.114980 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e83e8800-42f1-4f32-8b28-18dd2b81a68e admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:31.114980 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e83e8800-42f1-4f32-8b28-18dd2b81a68e admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:31.114980 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e83e8800-42f1-4f32-8b28-18dd2b81a68e admin admin] PATH_INFO: `/v3/projects/282fbaf5c62d42f894c4798e46ff5100/users/32408a0962f940938443225668963feb/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:31.136458 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e83e8800-42f1-4f32-8b28-18dd2b81a68e admin admin] RBAC: Authorizing `identity:create_grant(project_id=282fbaf5c62d42f894c4798e46ff5100, user_id=32408a0962f940938443225668963feb, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:31.139976 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e83e8800-42f1-4f32-8b28-18dd2b81a68e admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:31.163756 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1288/2589] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 01:02:31 2022] PUT /identity/v3/projects/282fbaf5c62d42f894c4798e46ff5100/users/32408a0962f940938443225668963feb/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 81 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 01:02:31.183448 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a4af33-7961-4288-9948-6a8e1278f4ee None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:31.184395 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a4af33-7961-4288-9948-6a8e1278f4ee None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:31.184395 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a4af33-7961-4288-9948-6a8e1278f4ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:31.215375 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:DELETE=6,keystone:SELECT=90,keystone:INSERT=4 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:02:31.232695 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-f8a4af33-7961-4288-9948-6a8e1278f4ee None None] MFA Rules not processed for user `32408a0962f940938443225668963feb`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:02:31.234939 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-f8a4af33-7961-4288-9948-6a8e1278f4ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:02:31.384466 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1302/2590] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:02:31 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:02:31.407707 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e83e8800-42f1-4f32-8b28-18dd2b81a68e admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:31.411036 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-e83e8800-42f1-4f32-8b28-18dd2b81a68e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:02:31.448574 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=89,keystone:DELETE=8,keystone:INSERT=3 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:02:31.596527 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e83e8800-42f1-4f32-8b28-18dd2b81a68e admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:31.604426 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635eff00-fd1b-4386-9f90-1554c9837f50 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:31.605878 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635eff00-fd1b-4386-9f90-1554c9837f50 service cinder] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:31.606553 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635eff00-fd1b-4386-9f90-1554c9837f50 service cinder] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:31.607090 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635eff00-fd1b-4386-9f90-1554c9837f50 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:31.635566 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-635eff00-fd1b-4386-9f90-1554c9837f50 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:31.635566 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-635eff00-fd1b-4386-9f90-1554c9837f50 service cinder] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:31.655684 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1289/2591] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:02:31 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 249 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:02:41.374412 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=28 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:02:41.643534 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=20 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:02:43.958614 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f27501-99f2-4bda-af28-2a5d956cd521 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:43.959260 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f27501-99f2-4bda-af28-2a5d956cd521 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:43.959638 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f27501-99f2-4bda-af28-2a5d956cd521 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:44.031357 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-07f27501-99f2-4bda-af28-2a5d956cd521 None None] MFA Rules not processed for user `3f20ffd8df994c1db1b1e7248db5b5a3`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:02:44.033077 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-07f27501-99f2-4bda-af28-2a5d956cd521 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:02:44.166262 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1303/2592] 10.222.0.52 () {58 vars in 967 bytes} [Sun Sep 25 01:02:43 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:02:44.182812 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635eff00-fd1b-4386-9f90-1554c9837f50 service cinder] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:44.184281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-635eff00-fd1b-4386-9f90-1554c9837f50 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:02:44.281500 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635eff00-fd1b-4386-9f90-1554c9837f50 service cinder] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:44.302291 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d2220f-47d4-4267-bd26-b0cce4748e05 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da606b3906dd4171bd76fd281e03e258', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:44.303906 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d2220f-47d4-4267-bd26-b0cce4748e05 service swift] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:44.304354 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d2220f-47d4-4267-bd26-b0cce4748e05 service swift] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:44.305623 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d2220f-47d4-4267-bd26-b0cce4748e05 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:44.319723 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03d2220f-47d4-4267-bd26-b0cce4748e05 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:44.321509 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03d2220f-47d4-4267-bd26-b0cce4748e05 service swift] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:44.340590 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1290/2593] 10.222.0.52 () {60 vars in 1384 bytes} [Sun Sep 25 01:02:44 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 158 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 25 01:02:52.818712 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f27501-99f2-4bda-af28-2a5d956cd521 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:52.917040 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f27501-99f2-4bda-af28-2a5d956cd521 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:52.935718 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19813d67-624b-46e3-924a-7eb522654f3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:52.936976 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19813d67-624b-46e3-924a-7eb522654f3e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:52.937648 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19813d67-624b-46e3-924a-7eb522654f3e admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:52.938259 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19813d67-624b-46e3-924a-7eb522654f3e admin admin] PATH_INFO: `/v3/users/32408a0962f940938443225668963feb` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:52.940677 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19813d67-624b-46e3-924a-7eb522654f3e admin admin] RBAC: Authorizing `identity:delete_user(user_id=32408a0962f940938443225668963feb)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:52.944204 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19813d67-624b-46e3-924a-7eb522654f3e admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:53.007707 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-19813d67-624b-46e3-924a-7eb522654f3e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '32408a0962f940938443225668963feb'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:53.008492 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-19813d67-624b-46e3-924a-7eb522654f3e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '32408a0962f940938443225668963feb'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:53.017688 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-19813d67-624b-46e3-924a-7eb522654f3e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '32408a0962f940938443225668963feb'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:53.018340 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-19813d67-624b-46e3-924a-7eb522654f3e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '32408a0962f940938443225668963feb'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:53.027904 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-19813d67-624b-46e3-924a-7eb522654f3e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '32408a0962f940938443225668963feb'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:53.050964 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1304/2594] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 01:02:52 2022] DELETE /identity/v3/users/32408a0962f940938443225668963feb => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:02:53.056209 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f3e5a9-807c-4981-b174-f7ff901c7364 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:53.056883 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f3e5a9-807c-4981-b174-f7ff901c7364 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:53.056883 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f3e5a9-807c-4981-b174-f7ff901c7364 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:53.080064 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-a7f3e5a9-807c-4981-b174-f7ff901c7364 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:02:53.082261 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-a7f3e5a9-807c-4981-b174-f7ff901c7364 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:02:53.182686 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1291/2595] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:02:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 01:02:53.193717 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19813d67-624b-46e3-924a-7eb522654f3e admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:53.195324 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-19813d67-624b-46e3-924a-7eb522654f3e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:02:53.238318 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:02:53.283902 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19813d67-624b-46e3-924a-7eb522654f3e admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:53.290480 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9669bce-fef2-4d1d-b6dc-25280c93c0de service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:53.291611 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9669bce-fef2-4d1d-b6dc-25280c93c0de service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:53.292095 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9669bce-fef2-4d1d-b6dc-25280c93c0de service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:53.292576 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9669bce-fef2-4d1d-b6dc-25280c93c0de service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:53.309551 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9669bce-fef2-4d1d-b6dc-25280c93c0de service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:53.312112 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9669bce-fef2-4d1d-b6dc-25280c93c0de service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:53.326541 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1305/2596] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:02:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 01:02:53.895850 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7f3e5a9-807c-4981-b174-f7ff901c7364 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:02:53.898405 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-a7f3e5a9-807c-4981-b174-f7ff901c7364 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:02:53.923888 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7f3e5a9-807c-4981-b174-f7ff901c7364 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:02:53.929057 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9fe75c1-42da-4cf7-9bd3-7514f3ce0103 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:02:53.929836 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9fe75c1-42da-4cf7-9bd3-7514f3ce0103 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:02:53.930179 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9fe75c1-42da-4cf7-9bd3-7514f3ce0103 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:02:53.930515 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9fe75c1-42da-4cf7-9bd3-7514f3ce0103 admin admin] PATH_INFO: `/v3/projects/282fbaf5c62d42f894c4798e46ff5100` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:02:53.931862 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9fe75c1-42da-4cf7-9bd3-7514f3ce0103 admin admin] RBAC: Authorizing `identity:delete_project(project_id=282fbaf5c62d42f894c4798e46ff5100)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:02:53.933878 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9fe75c1-42da-4cf7-9bd3-7514f3ce0103 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:02:53.993216 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-b9fe75c1-42da-4cf7-9bd3-7514f3ce0103 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '282fbaf5c62d42f894c4798e46ff5100', 'request_id': 'req-b9fe75c1-42da-4cf7-9bd3-7514f3ce0103'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:54.003035 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-b9fe75c1-42da-4cf7-9bd3-7514f3ce0103 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '282fbaf5c62d42f894c4798e46ff5100', 'request_id': 'req-b9fe75c1-42da-4cf7-9bd3-7514f3ce0103'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:54.012177 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-b9fe75c1-42da-4cf7-9bd3-7514f3ce0103 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '282fbaf5c62d42f894c4798e46ff5100', 'request_id': 'req-b9fe75c1-42da-4cf7-9bd3-7514f3ce0103'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:54.029667 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-b9fe75c1-42da-4cf7-9bd3-7514f3ce0103 admin admin] The token cache is being invalidate because project 282fbaf5c62d42f894c4798e46ff5100 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81386) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 01:02:54.030602 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-b9fe75c1-42da-4cf7-9bd3-7514f3ce0103 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:54.032062 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-b9fe75c1-42da-4cf7-9bd3-7514f3ce0103 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:02:54.034689 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1292/2597] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 01:02:53 2022] DELETE /identity/v3/projects/282fbaf5c62d42f894c4798e46ff5100 => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:03:03.321922 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=15 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:04.023293 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=86,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:04.340039 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9669bce-fef2-4d1d-b6dc-25280c93c0de service neutron] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:04.341928 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-a9669bce-fef2-4d1d-b6dc-25280c93c0de service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:04.355091 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9669bce-fef2-4d1d-b6dc-25280c93c0de service neutron] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:04.414888 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd739379-2263-40ad-9dc6-8faf8bc4efb7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:04.416217 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd739379-2263-40ad-9dc6-8faf8bc4efb7 service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:04.416699 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd739379-2263-40ad-9dc6-8faf8bc4efb7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:04.417075 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd739379-2263-40ad-9dc6-8faf8bc4efb7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:04.419440 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-cd739379-2263-40ad-9dc6-8faf8bc4efb7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:04.498744 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd739379-2263-40ad-9dc6-8faf8bc4efb7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:04.500135 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd739379-2263-40ad-9dc6-8faf8bc4efb7 service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:04.516425 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1306/2598] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:03:04 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 177 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 01:03:04.909769 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9fe75c1-42da-4cf7-9bd3-7514f3ce0103 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:04.911905 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-b9fe75c1-42da-4cf7-9bd3-7514f3ce0103 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:04.924933 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9fe75c1-42da-4cf7-9bd3-7514f3ce0103 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:04.929838 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbcf56a0-a29d-494a-b420-dfa9387b36df service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:04.930547 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcf56a0-a29d-494a-b420-dfa9387b36df service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:04.930876 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcf56a0-a29d-494a-b420-dfa9387b36df service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:04.931305 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcf56a0-a29d-494a-b420-dfa9387b36df service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:04.933523 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-dbcf56a0-a29d-494a-b420-dfa9387b36df service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:05.027125 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbcf56a0-a29d-494a-b420-dfa9387b36df service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:05.029309 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbcf56a0-a29d-494a-b420-dfa9387b36df service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:05.042593 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1293/2599] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:03:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:03:09.688595 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd739379-2263-40ad-9dc6-8faf8bc4efb7 service neutron] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:09.690745 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-cd739379-2263-40ad-9dc6-8faf8bc4efb7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:09.794112 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbcf56a0-a29d-494a-b420-dfa9387b36df service neutron] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:09.796920 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-dbcf56a0-a29d-494a-b420-dfa9387b36df service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:09.798484 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd739379-2263-40ad-9dc6-8faf8bc4efb7 service neutron] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:09.805284 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2292f28e-1e5a-4fe2-a43c-1d6a696c9c42 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '11a80b55dcd444df901c8bf80f2a2c26', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:09.806104 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2292f28e-1e5a-4fe2-a43c-1d6a696c9c42 service placement] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:09.806383 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2292f28e-1e5a-4fe2-a43c-1d6a696c9c42 service placement] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:09.806675 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2292f28e-1e5a-4fe2-a43c-1d6a696c9c42 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:09.808610 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-2292f28e-1e5a-4fe2-a43c-1d6a696c9c42 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:09.815992 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbcf56a0-a29d-494a-b420-dfa9387b36df service neutron] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:09.823040 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-006bd3a3-b640-43ac-b4b9-319fc57ae793 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:09.823719 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-006bd3a3-b640-43ac-b4b9-319fc57ae793 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:09.824036 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-006bd3a3-b640-43ac-b4b9-319fc57ae793 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:09.824497 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-006bd3a3-b640-43ac-b4b9-319fc57ae793 admin admin] PATH_INFO: `/v3/users/96386a16f0b54b7c94d32bd948caf118` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:09.825488 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2292f28e-1e5a-4fe2-a43c-1d6a696c9c42 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:09.825776 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-006bd3a3-b640-43ac-b4b9-319fc57ae793 admin admin] RBAC: Authorizing `identity:delete_user(user_id=96386a16f0b54b7c94d32bd948caf118)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:09.827120 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2292f28e-1e5a-4fe2-a43c-1d6a696c9c42 service placement] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:09.827709 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-006bd3a3-b640-43ac-b4b9-319fc57ae793 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:09.842038 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1307/2600] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:03:09 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:03:09.907111 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-006bd3a3-b640-43ac-b4b9-319fc57ae793 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '96386a16f0b54b7c94d32bd948caf118'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:09.916422 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-006bd3a3-b640-43ac-b4b9-319fc57ae793 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '96386a16f0b54b7c94d32bd948caf118'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:09.917474 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-006bd3a3-b640-43ac-b4b9-319fc57ae793 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '96386a16f0b54b7c94d32bd948caf118'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:09.925797 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-006bd3a3-b640-43ac-b4b9-319fc57ae793 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '96386a16f0b54b7c94d32bd948caf118'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:09.927354 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-006bd3a3-b640-43ac-b4b9-319fc57ae793 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '96386a16f0b54b7c94d32bd948caf118'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:09.947362 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1294/2601] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 01:03:09 2022] DELETE /identity/v3/users/96386a16f0b54b7c94d32bd948caf118 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:03:10.156864 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2292f28e-1e5a-4fe2-a43c-1d6a696c9c42 service placement] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:10.175431 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2292f28e-1e5a-4fe2-a43c-1d6a696c9c42 service placement] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:10.242738 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:10.269976 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d09e863a-f442-4881-8075-6d8ed9cfc83f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:10.270679 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09e863a-f442-4881-8075-6d8ed9cfc83f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:10.270989 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09e863a-f442-4881-8075-6d8ed9cfc83f admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:10.271269 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09e863a-f442-4881-8075-6d8ed9cfc83f admin admin] PATH_INFO: `/v3/projects/f08425c00f0c49ddbe42b21ddbc7ff3f` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:10.272775 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d09e863a-f442-4881-8075-6d8ed9cfc83f admin admin] RBAC: Authorizing `identity:delete_project(project_id=f08425c00f0c49ddbe42b21ddbc7ff3f)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:10.274737 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d09e863a-f442-4881-8075-6d8ed9cfc83f admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:10.340340 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-d09e863a-f442-4881-8075-6d8ed9cfc83f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f08425c00f0c49ddbe42b21ddbc7ff3f', 'request_id': 'req-d09e863a-f442-4881-8075-6d8ed9cfc83f'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:10.351937 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-d09e863a-f442-4881-8075-6d8ed9cfc83f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f08425c00f0c49ddbe42b21ddbc7ff3f', 'request_id': 'req-d09e863a-f442-4881-8075-6d8ed9cfc83f'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:10.359070 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-d09e863a-f442-4881-8075-6d8ed9cfc83f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f08425c00f0c49ddbe42b21ddbc7ff3f', 'request_id': 'req-d09e863a-f442-4881-8075-6d8ed9cfc83f'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:10.373193 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-d09e863a-f442-4881-8075-6d8ed9cfc83f admin admin] The token cache is being invalidate because project f08425c00f0c49ddbe42b21ddbc7ff3f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 01:03:10.373499 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-d09e863a-f442-4881-8075-6d8ed9cfc83f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:10.374203 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-d09e863a-f442-4881-8075-6d8ed9cfc83f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:10.375744 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1308/2602] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 01:03:10 2022] DELETE /identity/v3/projects/f08425c00f0c49ddbe42b21ddbc7ff3f => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:03:10.381627 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54923645-6aac-4f49-80b5-78634aeff300 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:10.381954 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54923645-6aac-4f49-80b5-78634aeff300 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:10.382166 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54923645-6aac-4f49-80b5-78634aeff300 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:10.404742 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-54923645-6aac-4f49-80b5-78634aeff300 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:03:10.407135 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-54923645-6aac-4f49-80b5-78634aeff300 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:10.493599 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1295/2603] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:03:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:03:10.501716 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4744636c-ced3-4170-a86e-7b2eda9a056e None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:10.502052 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4744636c-ced3-4170-a86e-7b2eda9a056e None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:10.502222 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4744636c-ced3-4170-a86e-7b2eda9a056e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:10.524775 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-4744636c-ced3-4170-a86e-7b2eda9a056e None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:03:10.526064 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-4744636c-ced3-4170-a86e-7b2eda9a056e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:10.535044 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1309/2604] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:03:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:03:10.540692 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54923645-6aac-4f49-80b5-78634aeff300 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:10.551396 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54923645-6aac-4f49-80b5-78634aeff300 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:10.556691 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4fc64d9-b0c4-49d1-ab92-f6df82813273 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:10.557362 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fc64d9-b0c4-49d1-ab92-f6df82813273 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:10.557802 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fc64d9-b0c4-49d1-ab92-f6df82813273 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:10.558079 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fc64d9-b0c4-49d1-ab92-f6df82813273 admin admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:10.558757 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4fc64d9-b0c4-49d1-ab92-f6df82813273 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 01:03:10.559062 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4fc64d9-b0c4-49d1-ab92-f6df82813273 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:10.560694 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4fc64d9-b0c4-49d1-ab92-f6df82813273 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:10.562498 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:10.576730 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1296/2605] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 01:03:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 01:03:10.583608 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4744636c-ced3-4170-a86e-7b2eda9a056e None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:10.594152 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4744636c-ced3-4170-a86e-7b2eda9a056e None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:10.599616 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e30f237d-83cb-4320-af18-a9ff3f1302f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:10.600232 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30f237d-83cb-4320-af18-a9ff3f1302f7 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:10.600504 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30f237d-83cb-4320-af18-a9ff3f1302f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:10.600769 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30f237d-83cb-4320-af18-a9ff3f1302f7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:10.601466 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e30f237d-83cb-4320-af18-a9ff3f1302f7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:10.603468 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e30f237d-83cb-4320-af18-a9ff3f1302f7 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:10.636601 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1310/2606] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 01:03:10 2022] POST /identity/v3/projects => generated 375 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 01:03:10.640986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4fc64d9-b0c4-49d1-ab92-f6df82813273 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:10.653045 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4fc64d9-b0c4-49d1-ab92-f6df82813273 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:10.730614 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9908d66-328a-4531-b2e5-094dd9576ae6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:10.731288 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9908d66-328a-4531-b2e5-094dd9576ae6 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:10.731607 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9908d66-328a-4531-b2e5-094dd9576ae6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:10.731921 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9908d66-328a-4531-b2e5-094dd9576ae6 admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:10.732697 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9908d66-328a-4531-b2e5-094dd9576ae6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:10.735676 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9908d66-328a-4531-b2e5-094dd9576ae6 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:10.761123 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1297/2607] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 01:03:10 2022] POST /identity/v3/users => generated 288 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 01:03:10.766840 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e30f237d-83cb-4320-af18-a9ff3f1302f7 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:10.779350 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e30f237d-83cb-4320-af18-a9ff3f1302f7 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:10.784656 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2c2c636-9175-41de-876d-10f3c84890ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:10.785418 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c2c636-9175-41de-876d-10f3c84890ea admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:10.785656 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c2c636-9175-41de-876d-10f3c84890ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:10.786542 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c2c636-9175-41de-876d-10f3c84890ea admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:10.787261 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2c2c636-9175-41de-876d-10f3c84890ea admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:10.788905 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2c2c636-9175-41de-876d-10f3c84890ea admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:10.800914 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1311/2608] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:03:10 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 01:03:10.806258 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9908d66-328a-4531-b2e5-094dd9576ae6 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:10.818845 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9908d66-328a-4531-b2e5-094dd9576ae6 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:10.824252 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a22dcce-abcc-4c97-9197-3eb65ec3aee7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:10.824879 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a22dcce-abcc-4c97-9197-3eb65ec3aee7 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:10.825193 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a22dcce-abcc-4c97-9197-3eb65ec3aee7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:10.825490 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a22dcce-abcc-4c97-9197-3eb65ec3aee7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:10.826144 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a22dcce-abcc-4c97-9197-3eb65ec3aee7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:10.827766 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a22dcce-abcc-4c97-9197-3eb65ec3aee7 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:10.841737 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1298/2609] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:03:10 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 01:03:10.846795 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2c2c636-9175-41de-876d-10f3c84890ea admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:10.858567 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2c2c636-9175-41de-876d-10f3c84890ea admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:10.864371 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe2f4a0-6eb6-4a1f-b88f-0816cd05b4ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:10.865088 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe2f4a0-6eb6-4a1f-b88f-0816cd05b4ff admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:10.865430 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe2f4a0-6eb6-4a1f-b88f-0816cd05b4ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:10.865679 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe2f4a0-6eb6-4a1f-b88f-0816cd05b4ff admin admin] PATH_INFO: `/v3/projects/3b39228a17314fa49ffb00c024e7dc68/users/43e16e8590f445348c9b3d121f66ecb9/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:10.882426 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe2f4a0-6eb6-4a1f-b88f-0816cd05b4ff admin admin] RBAC: Authorizing `identity:create_grant(project_id=3b39228a17314fa49ffb00c024e7dc68, user_id=43e16e8590f445348c9b3d121f66ecb9, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:10.885104 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe2f4a0-6eb6-4a1f-b88f-0816cd05b4ff admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:10.905935 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1312/2610] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 01:03:10 2022] PUT /identity/v3/projects/3b39228a17314fa49ffb00c024e7dc68/users/43e16e8590f445348c9b3d121f66ecb9/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:03:10.924865 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a8d896-7152-4f8a-998c-c8f90edb6cd3 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:10.925183 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a8d896-7152-4f8a-998c-c8f90edb6cd3 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:10.925571 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a8d896-7152-4f8a-998c-c8f90edb6cd3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:10.959183 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-26a8d896-7152-4f8a-998c-c8f90edb6cd3 None None] MFA Rules not processed for user `43e16e8590f445348c9b3d121f66ecb9`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:03:10.960411 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-26a8d896-7152-4f8a-998c-c8f90edb6cd3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:11.051216 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1299/2611] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:03:10 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:03:11.069012 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe2f4a0-6eb6-4a1f-b88f-0816cd05b4ff admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:11.071110 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-2fe2f4a0-6eb6-4a1f-b88f-0816cd05b4ff admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:11.156041 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:11.167693 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe2f4a0-6eb6-4a1f-b88f-0816cd05b4ff admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:11.179538 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0503d18c-32ca-4f64-af36-30abf957c272 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f142988a4dd14a5f96eacea45c046f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:11.180630 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0503d18c-32ca-4f64-af36-30abf957c272 service nova] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:11.181419 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0503d18c-32ca-4f64-af36-30abf957c272 service nova] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:11.181848 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0503d18c-32ca-4f64-af36-30abf957c272 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:11.199305 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0503d18c-32ca-4f64-af36-30abf957c272 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:11.203007 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0503d18c-32ca-4f64-af36-30abf957c272 service nova] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:11.222496 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1313/2612] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:03:11 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 01:03:11.511244 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a8d896-7152-4f8a-998c-c8f90edb6cd3 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:11.586084 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a8d896-7152-4f8a-998c-c8f90edb6cd3 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:11.591056 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c832e1a-79a1-4bc4-b0e7-94ebeda893f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:11.591710 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c832e1a-79a1-4bc4-b0e7-94ebeda893f5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:11.591986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c832e1a-79a1-4bc4-b0e7-94ebeda893f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:11.592255 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c832e1a-79a1-4bc4-b0e7-94ebeda893f5 admin admin] PATH_INFO: `/v3/users/43e16e8590f445348c9b3d121f66ecb9` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:11.593513 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c832e1a-79a1-4bc4-b0e7-94ebeda893f5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=43e16e8590f445348c9b3d121f66ecb9)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:11.595458 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c832e1a-79a1-4bc4-b0e7-94ebeda893f5 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:11.598355 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:11.655325 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-1c832e1a-79a1-4bc4-b0e7-94ebeda893f5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '43e16e8590f445348c9b3d121f66ecb9'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:11.666138 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-1c832e1a-79a1-4bc4-b0e7-94ebeda893f5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '43e16e8590f445348c9b3d121f66ecb9'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:11.666817 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-1c832e1a-79a1-4bc4-b0e7-94ebeda893f5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '43e16e8590f445348c9b3d121f66ecb9'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:11.675605 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-1c832e1a-79a1-4bc4-b0e7-94ebeda893f5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '43e16e8590f445348c9b3d121f66ecb9'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:11.677362 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-1c832e1a-79a1-4bc4-b0e7-94ebeda893f5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '43e16e8590f445348c9b3d121f66ecb9'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:11.701284 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1300/2613] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 01:03:11 2022] DELETE /identity/v3/users/43e16e8590f445348c9b3d121f66ecb9 => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:03:11.706421 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1086161-7f7b-407d-ba5b-020dfca67623 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:11.706750 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1086161-7f7b-407d-ba5b-020dfca67623 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:11.707069 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1086161-7f7b-407d-ba5b-020dfca67623 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:11.728985 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-e1086161-7f7b-407d-ba5b-020dfca67623 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:03:11.730259 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-e1086161-7f7b-407d-ba5b-020dfca67623 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:11.802443 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1314/2614] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:03:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 01:03:11.812634 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c832e1a-79a1-4bc4-b0e7-94ebeda893f5 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:11.814413 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-1c832e1a-79a1-4bc4-b0e7-94ebeda893f5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:11.883597 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c832e1a-79a1-4bc4-b0e7-94ebeda893f5 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:11.887521 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95fa2aa6-7bf6-4f60-bfa3-3fd8992f3de7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:11.888149 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95fa2aa6-7bf6-4f60-bfa3-3fd8992f3de7 service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:11.888431 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95fa2aa6-7bf6-4f60-bfa3-3fd8992f3de7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:11.888692 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95fa2aa6-7bf6-4f60-bfa3-3fd8992f3de7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:11.899565 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95fa2aa6-7bf6-4f60-bfa3-3fd8992f3de7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:11.901122 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95fa2aa6-7bf6-4f60-bfa3-3fd8992f3de7 service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:11.913060 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1301/2615] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:03:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 01:03:12.263375 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1086161-7f7b-407d-ba5b-020dfca67623 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:12.265181 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-e1086161-7f7b-407d-ba5b-020dfca67623 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:12.280305 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1086161-7f7b-407d-ba5b-020dfca67623 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:12.289611 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53bd06c7-109d-4455-b0a5-4670d93d0a06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:12.290702 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53bd06c7-109d-4455-b0a5-4670d93d0a06 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:12.291195 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53bd06c7-109d-4455-b0a5-4670d93d0a06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:12.291652 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53bd06c7-109d-4455-b0a5-4670d93d0a06 admin admin] PATH_INFO: `/v3/projects/3b39228a17314fa49ffb00c024e7dc68` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:12.293850 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53bd06c7-109d-4455-b0a5-4670d93d0a06 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3b39228a17314fa49ffb00c024e7dc68)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:12.297781 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53bd06c7-109d-4455-b0a5-4670d93d0a06 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:12.361477 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-53bd06c7-109d-4455-b0a5-4670d93d0a06 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3b39228a17314fa49ffb00c024e7dc68', 'request_id': 'req-53bd06c7-109d-4455-b0a5-4670d93d0a06'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:12.372441 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-53bd06c7-109d-4455-b0a5-4670d93d0a06 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3b39228a17314fa49ffb00c024e7dc68', 'request_id': 'req-53bd06c7-109d-4455-b0a5-4670d93d0a06'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:12.381211 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-53bd06c7-109d-4455-b0a5-4670d93d0a06 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3b39228a17314fa49ffb00c024e7dc68', 'request_id': 'req-53bd06c7-109d-4455-b0a5-4670d93d0a06'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:12.395025 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-53bd06c7-109d-4455-b0a5-4670d93d0a06 admin admin] The token cache is being invalidate because project 3b39228a17314fa49ffb00c024e7dc68 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 01:03:12.395522 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-53bd06c7-109d-4455-b0a5-4670d93d0a06 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:12.396348 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-53bd06c7-109d-4455-b0a5-4670d93d0a06 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:12.398219 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1315/2616] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 01:03:12 2022] DELETE /identity/v3/projects/3b39228a17314fa49ffb00c024e7dc68 => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:03:12.403684 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32027444-3525-4017-9896-34156c204f89 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:12.404007 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32027444-3525-4017-9896-34156c204f89 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:12.404321 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32027444-3525-4017-9896-34156c204f89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:12.427788 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-32027444-3525-4017-9896-34156c204f89 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:03:12.429247 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-32027444-3525-4017-9896-34156c204f89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:12.506113 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1302/2617] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:03:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 01:03:12.521900 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d5d3ca-c97d-471a-8983-31b0fa26923e None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:12.522721 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d5d3ca-c97d-471a-8983-31b0fa26923e None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:12.523313 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d5d3ca-c97d-471a-8983-31b0fa26923e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:12.551922 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-a8d5d3ca-c97d-471a-8983-31b0fa26923e None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:03:12.553378 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-a8d5d3ca-c97d-471a-8983-31b0fa26923e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:12.562091 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1316/2618] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:03:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:03:12.567803 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32027444-3525-4017-9896-34156c204f89 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:12.579249 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32027444-3525-4017-9896-34156c204f89 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:12.584469 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e498d116-994e-42af-8d3f-81bb27437993 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:12.585098 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e498d116-994e-42af-8d3f-81bb27437993 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:12.585422 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e498d116-994e-42af-8d3f-81bb27437993 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:12.585679 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e498d116-994e-42af-8d3f-81bb27437993 admin admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:12.586341 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e498d116-994e-42af-8d3f-81bb27437993 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 01:03:12.586642 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e498d116-994e-42af-8d3f-81bb27437993 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:12.588277 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e498d116-994e-42af-8d3f-81bb27437993 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:12.602765 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1303/2619] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 01:03:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 01:03:12.607395 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d5d3ca-c97d-471a-8983-31b0fa26923e None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:12.618440 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d5d3ca-c97d-471a-8983-31b0fa26923e None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:12.623668 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9296f72-1992-46ef-a53b-f3488b56d7fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:12.624283 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9296f72-1992-46ef-a53b-f3488b56d7fd admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:12.624565 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9296f72-1992-46ef-a53b-f3488b56d7fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:12.624824 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9296f72-1992-46ef-a53b-f3488b56d7fd admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:12.625516 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9296f72-1992-46ef-a53b-f3488b56d7fd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:12.627486 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9296f72-1992-46ef-a53b-f3488b56d7fd admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:12.670857 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1317/2620] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 01:03:12 2022] POST /identity/v3/projects => generated 381 bytes in 64 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 01:03:12.675496 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e498d116-994e-42af-8d3f-81bb27437993 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:12.687403 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e498d116-994e-42af-8d3f-81bb27437993 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:12.697150 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:12.758339 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d69591e-520f-4c1f-a415-1def1a664358 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:12.758871 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d69591e-520f-4c1f-a415-1def1a664358 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:12.759145 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d69591e-520f-4c1f-a415-1def1a664358 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:12.759411 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d69591e-520f-4c1f-a415-1def1a664358 admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:12.760076 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d69591e-520f-4c1f-a415-1def1a664358 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:12.762153 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d69591e-520f-4c1f-a415-1def1a664358 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:12.787496 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1304/2621] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 01:03:12 2022] POST /identity/v3/users => generated 291 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 01:03:12.794597 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9296f72-1992-46ef-a53b-f3488b56d7fd admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:12.817111 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9296f72-1992-46ef-a53b-f3488b56d7fd admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:12.826029 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6bc6aff-c06d-48cb-b8cf-b8b1291b9058 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:12.827075 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bc6aff-c06d-48cb-b8cf-b8b1291b9058 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:12.827711 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bc6aff-c06d-48cb-b8cf-b8b1291b9058 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:12.828164 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bc6aff-c06d-48cb-b8cf-b8b1291b9058 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:12.829405 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6bc6aff-c06d-48cb-b8cf-b8b1291b9058 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:12.834051 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6bc6aff-c06d-48cb-b8cf-b8b1291b9058 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:12.858978 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1318/2622] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:03:12 2022] GET /identity/v3/roles => generated 1420 bytes in 66 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 01:03:12.864245 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d69591e-520f-4c1f-a415-1def1a664358 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:12.875929 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d69591e-520f-4c1f-a415-1def1a664358 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:12.881882 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef6acbf-ca13-465f-b41f-f18d1f5cd340 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:12.882510 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef6acbf-ca13-465f-b41f-f18d1f5cd340 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:12.882800 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef6acbf-ca13-465f-b41f-f18d1f5cd340 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:12.883046 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef6acbf-ca13-465f-b41f-f18d1f5cd340 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:12.883694 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef6acbf-ca13-465f-b41f-f18d1f5cd340 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:12.885360 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef6acbf-ca13-465f-b41f-f18d1f5cd340 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:12.898694 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1305/2623] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:03:12 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 01:03:12.906457 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6bc6aff-c06d-48cb-b8cf-b8b1291b9058 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:12.926983 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6bc6aff-c06d-48cb-b8cf-b8b1291b9058 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:12.935102 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ae6625a-f8db-4ac6-9cb3-9bd5a64750d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:12.936200 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae6625a-f8db-4ac6-9cb3-9bd5a64750d5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:12.936692 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae6625a-f8db-4ac6-9cb3-9bd5a64750d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:12.937176 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae6625a-f8db-4ac6-9cb3-9bd5a64750d5 admin admin] PATH_INFO: `/v3/projects/70c044369bd84c5c92793c8b5a8842f2/users/601f93ba2f3d41c0b5f572a98d812da1/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:12.958880 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ae6625a-f8db-4ac6-9cb3-9bd5a64750d5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=70c044369bd84c5c92793c8b5a8842f2, user_id=601f93ba2f3d41c0b5f572a98d812da1, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:12.961874 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ae6625a-f8db-4ac6-9cb3-9bd5a64750d5 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:12.978597 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1319/2624] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 01:03:12 2022] PUT /identity/v3/projects/70c044369bd84c5c92793c8b5a8842f2/users/601f93ba2f3d41c0b5f572a98d812da1/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 74 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:03:12.991247 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121a38b7-7204-4d96-b722-c245939babd0 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:12.992030 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121a38b7-7204-4d96-b722-c245939babd0 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:12.992030 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121a38b7-7204-4d96-b722-c245939babd0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:13.024497 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-121a38b7-7204-4d96-b722-c245939babd0 None None] MFA Rules not processed for user `601f93ba2f3d41c0b5f572a98d812da1`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:03:13.025727 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-121a38b7-7204-4d96-b722-c245939babd0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:13.134319 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1306/2625] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:03:12 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:03:13.140116 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ae6625a-f8db-4ac6-9cb3-9bd5a64750d5 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:13.159073 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:13.220751 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ae6625a-f8db-4ac6-9cb3-9bd5a64750d5 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:13.226859 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df7ba60-bf63-4fa9-8bff-5deed1e694d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:13.227489 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df7ba60-bf63-4fa9-8bff-5deed1e694d6 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:13.227777 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df7ba60-bf63-4fa9-8bff-5deed1e694d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:13.228034 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df7ba60-bf63-4fa9-8bff-5deed1e694d6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:13.228759 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5df7ba60-bf63-4fa9-8bff-5deed1e694d6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:13.230947 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5df7ba60-bf63-4fa9-8bff-5deed1e694d6 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:13.267902 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1320/2626] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 01:03:13 2022] POST /identity/v3/projects => generated 379 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 01:03:13.273351 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-121a38b7-7204-4d96-b722-c245939babd0 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:13.287577 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-121a38b7-7204-4d96-b722-c245939babd0 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:13.380917 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3ee060-69d9-42d0-8f22-c9e8f62e3008 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:13.381309 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3ee060-69d9-42d0-8f22-c9e8f62e3008 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:13.381467 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3ee060-69d9-42d0-8f22-c9e8f62e3008 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:13.381771 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3ee060-69d9-42d0-8f22-c9e8f62e3008 admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:13.382385 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e3ee060-69d9-42d0-8f22-c9e8f62e3008 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:13.384593 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e3ee060-69d9-42d0-8f22-c9e8f62e3008 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:13.409862 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1307/2627] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 01:03:13 2022] POST /identity/v3/users => generated 296 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 01:03:13.414380 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df7ba60-bf63-4fa9-8bff-5deed1e694d6 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:13.425887 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df7ba60-bf63-4fa9-8bff-5deed1e694d6 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:13.433358 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a302ff4a-42b5-45f0-ac5c-fc872cefd884 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:13.434309 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a302ff4a-42b5-45f0-ac5c-fc872cefd884 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:13.434677 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a302ff4a-42b5-45f0-ac5c-fc872cefd884 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:13.435052 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a302ff4a-42b5-45f0-ac5c-fc872cefd884 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:13.435949 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a302ff4a-42b5-45f0-ac5c-fc872cefd884 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:13.438476 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a302ff4a-42b5-45f0-ac5c-fc872cefd884 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:13.453636 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1321/2628] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:03:13 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 01:03:13.458345 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3ee060-69d9-42d0-8f22-c9e8f62e3008 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:13.469542 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3ee060-69d9-42d0-8f22-c9e8f62e3008 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:13.475453 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-675b8e55-32b6-4f18-9cd4-3b168a8f287d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:13.477146 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-675b8e55-32b6-4f18-9cd4-3b168a8f287d admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:13.477476 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-675b8e55-32b6-4f18-9cd4-3b168a8f287d admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:13.477963 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-675b8e55-32b6-4f18-9cd4-3b168a8f287d admin admin] PATH_INFO: `/v3/projects/38ed0c63f8ff49348d59d8d5deb6dfd9/users/4b6dc84221364d58b9ab6e7f67612165/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:13.483874 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:13.498511 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-675b8e55-32b6-4f18-9cd4-3b168a8f287d admin admin] RBAC: Authorizing `identity:create_grant(project_id=38ed0c63f8ff49348d59d8d5deb6dfd9, user_id=4b6dc84221364d58b9ab6e7f67612165, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:13.501664 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-675b8e55-32b6-4f18-9cd4-3b168a8f287d admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:13.520137 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1308/2629] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 01:03:13 2022] PUT /identity/v3/projects/38ed0c63f8ff49348d59d8d5deb6dfd9/users/4b6dc84221364d58b9ab6e7f67612165/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:03:13.524593 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a302ff4a-42b5-45f0-ac5c-fc872cefd884 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:13.538506 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a302ff4a-42b5-45f0-ac5c-fc872cefd884 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:13.612137 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba590b6-40d2-4d6f-ac87-87ec7605b388 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:13.612758 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba590b6-40d2-4d6f-ac87-87ec7605b388 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:13.613032 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba590b6-40d2-4d6f-ac87-87ec7605b388 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:13.613401 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba590b6-40d2-4d6f-ac87-87ec7605b388 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:13.614678 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ba590b6-40d2-4d6f-ac87-87ec7605b388 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:13.616282 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ba590b6-40d2-4d6f-ac87-87ec7605b388 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:13.627847 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1322/2630] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:03:13 2022] GET /identity/v3/roles => generated 1420 bytes in 104 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 01:03:13.632869 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-675b8e55-32b6-4f18-9cd4-3b168a8f287d admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:13.648872 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-675b8e55-32b6-4f18-9cd4-3b168a8f287d admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:13.654104 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0d09db-bafe-43de-8950-5142edbaa398 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:13.654900 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0d09db-bafe-43de-8950-5142edbaa398 admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:13.655260 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0d09db-bafe-43de-8950-5142edbaa398 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:13.655670 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0d09db-bafe-43de-8950-5142edbaa398 admin admin] PATH_INFO: `/v3/projects/38ed0c63f8ff49348d59d8d5deb6dfd9/users/4b6dc84221364d58b9ab6e7f67612165/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:13.658089 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f0d09db-bafe-43de-8950-5142edbaa398 admin admin] RBAC: Authorizing `identity:create_grant(project_id=38ed0c63f8ff49348d59d8d5deb6dfd9, user_id=4b6dc84221364d58b9ab6e7f67612165, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:13.661939 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f0d09db-bafe-43de-8950-5142edbaa398 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:13.675525 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1309/2631] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 01:03:13 2022] PUT /identity/v3/projects/38ed0c63f8ff49348d59d8d5deb6dfd9/users/4b6dc84221364d58b9ab6e7f67612165/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 44 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:03:13.690278 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5953e2c0-05fe-4933-8ac7-34f2dcceaebb None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:13.690769 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5953e2c0-05fe-4933-8ac7-34f2dcceaebb None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:13.691234 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5953e2c0-05fe-4933-8ac7-34f2dcceaebb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:13.731693 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-5953e2c0-05fe-4933-8ac7-34f2dcceaebb None None] MFA Rules not processed for user `4b6dc84221364d58b9ab6e7f67612165`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:03:13.732958 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-5953e2c0-05fe-4933-8ac7-34f2dcceaebb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:13.824556 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1323/2632] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:03:13 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:03:13.836899 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0d09db-bafe-43de-8950-5142edbaa398 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:13.838681 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-2f0d09db-bafe-43de-8950-5142edbaa398 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:13.916557 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0d09db-bafe-43de-8950-5142edbaa398 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:13.920765 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8827b215-b815-46e9-83af-d9a56cc6a7d5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:13.921434 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8827b215-b815-46e9-83af-d9a56cc6a7d5 service cinder] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:13.921739 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8827b215-b815-46e9-83af-d9a56cc6a7d5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:13.921953 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8827b215-b815-46e9-83af-d9a56cc6a7d5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:13.934802 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8827b215-b815-46e9-83af-d9a56cc6a7d5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:13.937324 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8827b215-b815-46e9-83af-d9a56cc6a7d5 service cinder] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:13.956502 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1310/2633] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:03:13 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:03:14.075392 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5953e2c0-05fe-4933-8ac7-34f2dcceaebb None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:14.087507 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5953e2c0-05fe-4933-8ac7-34f2dcceaebb None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:14.092990 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c59123b0-d7f9-4ae3-8d78-71adc55a9580 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:14.093876 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59123b0-d7f9-4ae3-8d78-71adc55a9580 service cinder] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:14.094215 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59123b0-d7f9-4ae3-8d78-71adc55a9580 service cinder] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:14.094534 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59123b0-d7f9-4ae3-8d78-71adc55a9580 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:14.100473 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:14.170045 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c59123b0-d7f9-4ae3-8d78-71adc55a9580 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:14.171534 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c59123b0-d7f9-4ae3-8d78-71adc55a9580 service cinder] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:14.185829 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1324/2634] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:03:14 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 01:03:23.619577 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8827b215-b815-46e9-83af-d9a56cc6a7d5 service cinder] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:23.699163 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8827b215-b815-46e9-83af-d9a56cc6a7d5 service cinder] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:23.705886 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb4fb6a7-226c-4b69-bc2b-8aa6f78e26ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:23.706663 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb4fb6a7-226c-4b69-bc2b-8aa6f78e26ca admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:23.707056 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb4fb6a7-226c-4b69-bc2b-8aa6f78e26ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:23.707415 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb4fb6a7-226c-4b69-bc2b-8aa6f78e26ca admin admin] PATH_INFO: `/v3/users/601f93ba2f3d41c0b5f572a98d812da1` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:23.708898 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb4fb6a7-226c-4b69-bc2b-8aa6f78e26ca admin admin] RBAC: Authorizing `identity:delete_user(user_id=601f93ba2f3d41c0b5f572a98d812da1)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:23.711105 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb4fb6a7-226c-4b69-bc2b-8aa6f78e26ca admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:23.781707 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-eb4fb6a7-226c-4b69-bc2b-8aa6f78e26ca admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '601f93ba2f3d41c0b5f572a98d812da1'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:23.794993 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-eb4fb6a7-226c-4b69-bc2b-8aa6f78e26ca admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '601f93ba2f3d41c0b5f572a98d812da1'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:23.795875 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-eb4fb6a7-226c-4b69-bc2b-8aa6f78e26ca admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '601f93ba2f3d41c0b5f572a98d812da1'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:23.810897 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-eb4fb6a7-226c-4b69-bc2b-8aa6f78e26ca admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '601f93ba2f3d41c0b5f572a98d812da1'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:23.812010 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-eb4fb6a7-226c-4b69-bc2b-8aa6f78e26ca admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '601f93ba2f3d41c0b5f572a98d812da1'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:23.832042 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=88,keystone:INSERT=3,keystone:DELETE=9 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:23.842520 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1311/2635] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 01:03:23 2022] DELETE /identity/v3/users/601f93ba2f3d41c0b5f572a98d812da1 => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:03:23.849747 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf54d06c-db1c-4cee-91d5-249f6566dc71 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:23.850479 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf54d06c-db1c-4cee-91d5-249f6566dc71 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:23.850479 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf54d06c-db1c-4cee-91d5-249f6566dc71 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:23.872734 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-cf54d06c-db1c-4cee-91d5-249f6566dc71 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:03:23.873925 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-cf54d06c-db1c-4cee-91d5-249f6566dc71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:23.950156 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1325/2636] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:03:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 01:03:23.961294 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb4fb6a7-226c-4b69-bc2b-8aa6f78e26ca admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:23.962771 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-eb4fb6a7-226c-4b69-bc2b-8aa6f78e26ca admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:24.034765 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb4fb6a7-226c-4b69-bc2b-8aa6f78e26ca admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:24.039090 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd370ed9-9158-48f9-a92d-901432090c81 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:24.039725 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd370ed9-9158-48f9-a92d-901432090c81 service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:24.040035 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd370ed9-9158-48f9-a92d-901432090c81 service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:24.040363 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd370ed9-9158-48f9-a92d-901432090c81 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:24.054097 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd370ed9-9158-48f9-a92d-901432090c81 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:24.055647 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd370ed9-9158-48f9-a92d-901432090c81 service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:24.069063 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1312/2637] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:03:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:03:24.469936 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf54d06c-db1c-4cee-91d5-249f6566dc71 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:24.473572 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-cf54d06c-db1c-4cee-91d5-249f6566dc71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:24.492884 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf54d06c-db1c-4cee-91d5-249f6566dc71 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:24.508722 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de03da8-8782-4a69-9891-03360424fb37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:24.509586 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de03da8-8782-4a69-9891-03360424fb37 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:24.509990 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de03da8-8782-4a69-9891-03360424fb37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:24.510359 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de03da8-8782-4a69-9891-03360424fb37 admin admin] PATH_INFO: `/v3/projects/70c044369bd84c5c92793c8b5a8842f2` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:24.512143 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5de03da8-8782-4a69-9891-03360424fb37 admin admin] RBAC: Authorizing `identity:delete_project(project_id=70c044369bd84c5c92793c8b5a8842f2)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:24.515497 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5de03da8-8782-4a69-9891-03360424fb37 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:24.611220 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-5de03da8-8782-4a69-9891-03360424fb37 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '70c044369bd84c5c92793c8b5a8842f2', 'request_id': 'req-5de03da8-8782-4a69-9891-03360424fb37'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:24.624857 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-5de03da8-8782-4a69-9891-03360424fb37 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '70c044369bd84c5c92793c8b5a8842f2', 'request_id': 'req-5de03da8-8782-4a69-9891-03360424fb37'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:24.638537 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-5de03da8-8782-4a69-9891-03360424fb37 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '70c044369bd84c5c92793c8b5a8842f2', 'request_id': 'req-5de03da8-8782-4a69-9891-03360424fb37'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:24.660087 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-5de03da8-8782-4a69-9891-03360424fb37 admin admin] The token cache is being invalidate because project 70c044369bd84c5c92793c8b5a8842f2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 01:03:24.661395 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-5de03da8-8782-4a69-9891-03360424fb37 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:24.662203 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-5de03da8-8782-4a69-9891-03360424fb37 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:24.663868 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1326/2638] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 01:03:24 2022] DELETE /identity/v3/projects/70c044369bd84c5c92793c8b5a8842f2 => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:03:24.670703 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd370ed9-9158-48f9-a92d-901432090c81 service neutron] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:24.688951 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd370ed9-9158-48f9-a92d-901432090c81 service neutron] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:24.795758 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edc56b0e-630d-4139-a75f-d14feb754970 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:24.796544 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc56b0e-630d-4139-a75f-d14feb754970 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:24.796934 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc56b0e-630d-4139-a75f-d14feb754970 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:24.797339 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc56b0e-630d-4139-a75f-d14feb754970 admin admin] PATH_INFO: `/v3/users/4b6dc84221364d58b9ab6e7f67612165` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:24.799220 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edc56b0e-630d-4139-a75f-d14feb754970 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4b6dc84221364d58b9ab6e7f67612165)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:24.803500 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edc56b0e-630d-4139-a75f-d14feb754970 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:24.895076 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-edc56b0e-630d-4139-a75f-d14feb754970 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4b6dc84221364d58b9ab6e7f67612165'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:24.912625 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-edc56b0e-630d-4139-a75f-d14feb754970 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4b6dc84221364d58b9ab6e7f67612165'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:24.913232 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-edc56b0e-630d-4139-a75f-d14feb754970 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4b6dc84221364d58b9ab6e7f67612165'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:24.922220 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-edc56b0e-630d-4139-a75f-d14feb754970 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4b6dc84221364d58b9ab6e7f67612165'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:24.923064 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-edc56b0e-630d-4139-a75f-d14feb754970 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4b6dc84221364d58b9ab6e7f67612165'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:24.944529 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1313/2639] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 01:03:24 2022] DELETE /identity/v3/users/4b6dc84221364d58b9ab6e7f67612165 => generated 0 bytes in 275 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:03:25.369417 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de03da8-8782-4a69-9891-03360424fb37 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:25.371673 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-5de03da8-8782-4a69-9891-03360424fb37 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:25.394064 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de03da8-8782-4a69-9891-03360424fb37 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:25.435986 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:25.486661 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4437cc52-79f5-4674-b6f7-7bb28de1ce4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:25.487737 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4437cc52-79f5-4674-b6f7-7bb28de1ce4a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:25.491525 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4437cc52-79f5-4674-b6f7-7bb28de1ce4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:25.491981 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4437cc52-79f5-4674-b6f7-7bb28de1ce4a admin admin] PATH_INFO: `/v3/projects/38ed0c63f8ff49348d59d8d5deb6dfd9` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:25.493596 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4437cc52-79f5-4674-b6f7-7bb28de1ce4a admin admin] RBAC: Authorizing `identity:delete_project(project_id=38ed0c63f8ff49348d59d8d5deb6dfd9)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:25.496018 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4437cc52-79f5-4674-b6f7-7bb28de1ce4a admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:25.566458 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-4437cc52-79f5-4674-b6f7-7bb28de1ce4a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '38ed0c63f8ff49348d59d8d5deb6dfd9', 'request_id': 'req-4437cc52-79f5-4674-b6f7-7bb28de1ce4a'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:25.575972 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-4437cc52-79f5-4674-b6f7-7bb28de1ce4a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '38ed0c63f8ff49348d59d8d5deb6dfd9', 'request_id': 'req-4437cc52-79f5-4674-b6f7-7bb28de1ce4a'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:25.583602 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-4437cc52-79f5-4674-b6f7-7bb28de1ce4a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '38ed0c63f8ff49348d59d8d5deb6dfd9', 'request_id': 'req-4437cc52-79f5-4674-b6f7-7bb28de1ce4a'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:25.595790 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-4437cc52-79f5-4674-b6f7-7bb28de1ce4a admin admin] The token cache is being invalidate because project 38ed0c63f8ff49348d59d8d5deb6dfd9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 01:03:25.596075 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-4437cc52-79f5-4674-b6f7-7bb28de1ce4a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:25.596792 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-4437cc52-79f5-4674-b6f7-7bb28de1ce4a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:25.598435 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1327/2640] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 01:03:25 2022] DELETE /identity/v3/projects/38ed0c63f8ff49348d59d8d5deb6dfd9 => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:03:25.603904 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7817a9-8545-48b6-90fe-3517442924c6 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:25.604311 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7817a9-8545-48b6-90fe-3517442924c6 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:25.604756 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7817a9-8545-48b6-90fe-3517442924c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:25.632189 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-9e7817a9-8545-48b6-90fe-3517442924c6 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:03:25.633395 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-9e7817a9-8545-48b6-90fe-3517442924c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:25.645130 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:DELETE=11,keystone:SELECT=88,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:25.714663 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1314/2641] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:03:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 01:03:25.724913 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e50903-dd0c-42cf-b00f-45b1c1db12ab None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:25.725196 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e50903-dd0c-42cf-b00f-45b1c1db12ab None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:25.725499 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e50903-dd0c-42cf-b00f-45b1c1db12ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:25.750137 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-e9e50903-dd0c-42cf-b00f-45b1c1db12ab None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:03:25.751530 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-e9e50903-dd0c-42cf-b00f-45b1c1db12ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:25.763990 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1328/2642] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:03:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:03:25.768739 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7817a9-8545-48b6-90fe-3517442924c6 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:25.781129 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7817a9-8545-48b6-90fe-3517442924c6 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:25.786351 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f4675f-27ee-4577-a03f-ee58f7f0d032 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:25.787168 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f4675f-27ee-4577-a03f-ee58f7f0d032 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:25.787454 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f4675f-27ee-4577-a03f-ee58f7f0d032 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:25.787856 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f4675f-27ee-4577-a03f-ee58f7f0d032 admin admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:25.788569 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f4675f-27ee-4577-a03f-ee58f7f0d032 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 01:03:25.788919 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f4675f-27ee-4577-a03f-ee58f7f0d032 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:25.790872 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f4675f-27ee-4577-a03f-ee58f7f0d032 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:25.805924 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1315/2643] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 01:03:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 01:03:25.811225 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e50903-dd0c-42cf-b00f-45b1c1db12ab None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:25.823039 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e50903-dd0c-42cf-b00f-45b1c1db12ab None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:25.829735 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a526bb-bddc-4f91-a7c1-3c0192ea84c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:25.830512 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a526bb-bddc-4f91-a7c1-3c0192ea84c9 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:25.830830 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a526bb-bddc-4f91-a7c1-3c0192ea84c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:25.831136 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a526bb-bddc-4f91-a7c1-3c0192ea84c9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:25.832553 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3a526bb-bddc-4f91-a7c1-3c0192ea84c9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:25.836468 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3a526bb-bddc-4f91-a7c1-3c0192ea84c9 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:25.877605 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1329/2644] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 01:03:25 2022] POST /identity/v3/projects => generated 363 bytes in 67 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 01:03:25.883221 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f4675f-27ee-4577-a03f-ee58f7f0d032 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:25.895550 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f4675f-27ee-4577-a03f-ee58f7f0d032 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:25.970059 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab5abb7-f3c2-4052-bc22-c6ac316c348c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:25.970683 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab5abb7-f3c2-4052-bc22-c6ac316c348c admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:25.970959 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab5abb7-f3c2-4052-bc22-c6ac316c348c admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:25.971227 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab5abb7-f3c2-4052-bc22-c6ac316c348c admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:25.971921 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ab5abb7-f3c2-4052-bc22-c6ac316c348c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:25.974524 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ab5abb7-f3c2-4052-bc22-c6ac316c348c admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:26.001060 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1316/2645] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 01:03:25 2022] POST /identity/v3/users => generated 282 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 01:03:26.008201 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a526bb-bddc-4f91-a7c1-3c0192ea84c9 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:26.020542 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a526bb-bddc-4f91-a7c1-3c0192ea84c9 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:26.026351 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c750290b-535a-4348-9f22-3ad7dff5c7a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:26.027093 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c750290b-535a-4348-9f22-3ad7dff5c7a0 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:26.027379 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c750290b-535a-4348-9f22-3ad7dff5c7a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:26.027650 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c750290b-535a-4348-9f22-3ad7dff5c7a0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:26.028337 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c750290b-535a-4348-9f22-3ad7dff5c7a0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:26.030061 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c750290b-535a-4348-9f22-3ad7dff5c7a0 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:26.043269 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1330/2646] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:03:26 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 01:03:26.050167 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab5abb7-f3c2-4052-bc22-c6ac316c348c admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:26.069504 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab5abb7-f3c2-4052-bc22-c6ac316c348c admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:26.077544 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8141c80-08c4-4645-a92c-ae6930e4b417 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:26.078596 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8141c80-08c4-4645-a92c-ae6930e4b417 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:26.079081 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8141c80-08c4-4645-a92c-ae6930e4b417 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:26.079562 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8141c80-08c4-4645-a92c-ae6930e4b417 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:26.080765 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8141c80-08c4-4645-a92c-ae6930e4b417 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:26.083443 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8141c80-08c4-4645-a92c-ae6930e4b417 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:26.105362 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1317/2647] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:03:26 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 01:03:26.111830 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c750290b-535a-4348-9f22-3ad7dff5c7a0 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:26.124594 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c750290b-535a-4348-9f22-3ad7dff5c7a0 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:26.131231 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe7c7eb-14df-4e76-a548-e322b908deb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:26.132058 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe7c7eb-14df-4e76-a548-e322b908deb8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:26.132431 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe7c7eb-14df-4e76-a548-e322b908deb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:26.132839 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe7c7eb-14df-4e76-a548-e322b908deb8 admin admin] PATH_INFO: `/v3/projects/3097e5a686514e4ab558b5941f6efe43/users/bc4b996522cd40069f5a1d06b99c3ec5/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:26.156671 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe7c7eb-14df-4e76-a548-e322b908deb8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3097e5a686514e4ab558b5941f6efe43, user_id=bc4b996522cd40069f5a1d06b99c3ec5, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:26.159460 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe7c7eb-14df-4e76-a548-e322b908deb8 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:26.179783 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1331/2648] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 01:03:26 2022] PUT /identity/v3/projects/3097e5a686514e4ab558b5941f6efe43/users/bc4b996522cd40069f5a1d06b99c3ec5/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:03:26.193339 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c51db7-bffc-4df7-84b7-94b47ec8243f None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:26.193516 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c51db7-bffc-4df7-84b7-94b47ec8243f None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:26.193889 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c51db7-bffc-4df7-84b7-94b47ec8243f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:26.231284 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-a4c51db7-bffc-4df7-84b7-94b47ec8243f None None] MFA Rules not processed for user `bc4b996522cd40069f5a1d06b99c3ec5`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:03:26.232340 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-a4c51db7-bffc-4df7-84b7-94b47ec8243f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:26.324392 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1318/2649] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:03:26 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:03:26.333518 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe7c7eb-14df-4e76-a548-e322b908deb8 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:26.408054 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:26.415420 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe7c7eb-14df-4e76-a548-e322b908deb8 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:26.425900 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-248dce6f-c798-40e2-b2de-626d361b4476 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:26.426558 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248dce6f-c798-40e2-b2de-626d361b4476 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:26.426829 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248dce6f-c798-40e2-b2de-626d361b4476 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:26.427093 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248dce6f-c798-40e2-b2de-626d361b4476 admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:26.427750 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-248dce6f-c798-40e2-b2de-626d361b4476 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:26.429733 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-248dce6f-c798-40e2-b2de-626d361b4476 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:26.479477 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1332/2650] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 01:03:26 2022] POST /identity/v3/projects => generated 367 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 01:03:26.487228 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4c51db7-bffc-4df7-84b7-94b47ec8243f None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:26.505899 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4c51db7-bffc-4df7-84b7-94b47ec8243f None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:26.516228 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:26.585096 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d2e0b3a-a663-4e82-8470-1e5163e78301 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:26.586082 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d2e0b3a-a663-4e82-8470-1e5163e78301 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:26.586532 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d2e0b3a-a663-4e82-8470-1e5163e78301 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:26.586901 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d2e0b3a-a663-4e82-8470-1e5163e78301 admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:26.587669 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d2e0b3a-a663-4e82-8470-1e5163e78301 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:26.589837 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d2e0b3a-a663-4e82-8470-1e5163e78301 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:26.622734 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1319/2651] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 01:03:26 2022] POST /identity/v3/users => generated 290 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 01:03:26.624168 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-248dce6f-c798-40e2-b2de-626d361b4476 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:26.636942 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-248dce6f-c798-40e2-b2de-626d361b4476 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:26.642515 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b7bffc9-c6ae-4d8d-9b8b-e6a7a1340c67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:26.643467 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7bffc9-c6ae-4d8d-9b8b-e6a7a1340c67 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:26.643901 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7bffc9-c6ae-4d8d-9b8b-e6a7a1340c67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:26.644371 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7bffc9-c6ae-4d8d-9b8b-e6a7a1340c67 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:26.645630 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b7bffc9-c6ae-4d8d-9b8b-e6a7a1340c67 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:26.647585 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b7bffc9-c6ae-4d8d-9b8b-e6a7a1340c67 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:26.667054 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1333/2652] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:03:26 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 01:03:26.671566 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d2e0b3a-a663-4e82-8470-1e5163e78301 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:26.684024 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d2e0b3a-a663-4e82-8470-1e5163e78301 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:26.689317 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d92760-5c98-437f-83bf-c9455d84e33a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:26.690489 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d92760-5c98-437f-83bf-c9455d84e33a admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:26.690885 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d92760-5c98-437f-83bf-c9455d84e33a admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:26.691989 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d92760-5c98-437f-83bf-c9455d84e33a admin admin] PATH_INFO: `/v3/projects/954dbc1f2fb246f98a56ee695b836bc2/users/d2aa47c8c2fd4164a4ffe5b9690e561b/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:26.714609 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0d92760-5c98-437f-83bf-c9455d84e33a admin admin] RBAC: Authorizing `identity:create_grant(project_id=954dbc1f2fb246f98a56ee695b836bc2, user_id=d2aa47c8c2fd4164a4ffe5b9690e561b, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:26.718000 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0d92760-5c98-437f-83bf-c9455d84e33a admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:26.738087 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1320/2653] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 01:03:26 2022] PUT /identity/v3/projects/954dbc1f2fb246f98a56ee695b836bc2/users/d2aa47c8c2fd4164a4ffe5b9690e561b/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 01:03:26.742850 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b7bffc9-c6ae-4d8d-9b8b-e6a7a1340c67 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:26.758440 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b7bffc9-c6ae-4d8d-9b8b-e6a7a1340c67 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:26.842580 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c87b999-0ffd-4375-a4f7-c1ff7d509a44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:26.843507 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c87b999-0ffd-4375-a4f7-c1ff7d509a44 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:26.843844 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c87b999-0ffd-4375-a4f7-c1ff7d509a44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:26.844282 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c87b999-0ffd-4375-a4f7-c1ff7d509a44 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:26.845680 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c87b999-0ffd-4375-a4f7-c1ff7d509a44 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:26.848277 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c87b999-0ffd-4375-a4f7-c1ff7d509a44 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:26.862094 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1334/2654] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:03:26 2022] GET /identity/v3/roles => generated 1420 bytes in 120 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 01:03:26.867395 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d92760-5c98-437f-83bf-c9455d84e33a admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:26.879434 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d92760-5c98-437f-83bf-c9455d84e33a admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:26.885405 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b26c7a13-b9fc-4084-9a8c-71d6777c78b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:26.886080 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b26c7a13-b9fc-4084-9a8c-71d6777c78b0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:26.886352 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b26c7a13-b9fc-4084-9a8c-71d6777c78b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:26.886615 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b26c7a13-b9fc-4084-9a8c-71d6777c78b0 admin admin] PATH_INFO: `/v3/projects/954dbc1f2fb246f98a56ee695b836bc2/users/d2aa47c8c2fd4164a4ffe5b9690e561b/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:26.888637 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b26c7a13-b9fc-4084-9a8c-71d6777c78b0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=954dbc1f2fb246f98a56ee695b836bc2, user_id=d2aa47c8c2fd4164a4ffe5b9690e561b, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:26.891377 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b26c7a13-b9fc-4084-9a8c-71d6777c78b0 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:26.904672 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1321/2655] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 01:03:26 2022] PUT /identity/v3/projects/954dbc1f2fb246f98a56ee695b836bc2/users/d2aa47c8c2fd4164a4ffe5b9690e561b/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:03:26.921545 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c0963a-1a6c-4551-8d8b-26593a0e63c1 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:26.921842 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c0963a-1a6c-4551-8d8b-26593a0e63c1 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:26.922266 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c0963a-1a6c-4551-8d8b-26593a0e63c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:26.957169 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-b5c0963a-1a6c-4551-8d8b-26593a0e63c1 None None] MFA Rules not processed for user `d2aa47c8c2fd4164a4ffe5b9690e561b`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:03:26.959274 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-b5c0963a-1a6c-4551-8d8b-26593a0e63c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:27.064129 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1335/2656] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:03:26 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:03:27.077769 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b26c7a13-b9fc-4084-9a8c-71d6777c78b0 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:27.080565 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-b26c7a13-b9fc-4084-9a8c-71d6777c78b0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:27.178306 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b26c7a13-b9fc-4084-9a8c-71d6777c78b0 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:27.183065 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73eea12-525f-4ae8-901e-05ce4eab6f83 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:27.183827 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73eea12-525f-4ae8-901e-05ce4eab6f83 service cinder] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:27.184273 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73eea12-525f-4ae8-901e-05ce4eab6f83 service cinder] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:27.184639 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73eea12-525f-4ae8-901e-05ce4eab6f83 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:27.200364 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f73eea12-525f-4ae8-901e-05ce4eab6f83 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:27.202294 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f73eea12-525f-4ae8-901e-05ce4eab6f83 service cinder] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:27.214441 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1322/2657] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:03:27 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:03:27.280899 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c0963a-1a6c-4551-8d8b-26593a0e63c1 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:27.282778 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-b5c0963a-1a6c-4551-8d8b-26593a0e63c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:27.306134 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c0963a-1a6c-4551-8d8b-26593a0e63c1 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:27.313317 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2bf774c-3e37-486c-be2c-39d2ee6d7445 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:27.313317 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bf774c-3e37-486c-be2c-39d2ee6d7445 service cinder] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:27.313835 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bf774c-3e37-486c-be2c-39d2ee6d7445 service cinder] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:27.314315 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bf774c-3e37-486c-be2c-39d2ee6d7445 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:27.402503 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:27.422679 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2bf774c-3e37-486c-be2c-39d2ee6d7445 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:27.424189 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2bf774c-3e37-486c-be2c-39d2ee6d7445 service cinder] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:27.441645 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1336/2658] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:03:27 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 161 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:03:31.529033 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73eea12-525f-4ae8-901e-05ce4eab6f83 service cinder] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:31.619560 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73eea12-525f-4ae8-901e-05ce4eab6f83 service cinder] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:31.625485 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66bf7ae7-77a4-4605-a85a-74bcc64d6059 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:31.626145 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66bf7ae7-77a4-4605-a85a-74bcc64d6059 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:31.626420 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66bf7ae7-77a4-4605-a85a-74bcc64d6059 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:31.626695 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66bf7ae7-77a4-4605-a85a-74bcc64d6059 admin admin] PATH_INFO: `/v3/users/bc4b996522cd40069f5a1d06b99c3ec5` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:31.628037 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66bf7ae7-77a4-4605-a85a-74bcc64d6059 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bc4b996522cd40069f5a1d06b99c3ec5)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:31.631906 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66bf7ae7-77a4-4605-a85a-74bcc64d6059 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:31.640046 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:31.701865 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-66bf7ae7-77a4-4605-a85a-74bcc64d6059 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bc4b996522cd40069f5a1d06b99c3ec5'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:31.713844 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-66bf7ae7-77a4-4605-a85a-74bcc64d6059 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bc4b996522cd40069f5a1d06b99c3ec5'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:31.714699 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-66bf7ae7-77a4-4605-a85a-74bcc64d6059 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bc4b996522cd40069f5a1d06b99c3ec5'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:31.724950 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-66bf7ae7-77a4-4605-a85a-74bcc64d6059 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bc4b996522cd40069f5a1d06b99c3ec5'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:31.726033 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-66bf7ae7-77a4-4605-a85a-74bcc64d6059 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bc4b996522cd40069f5a1d06b99c3ec5'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:31.749572 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1323/2659] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 01:03:31 2022] DELETE /identity/v3/users/bc4b996522cd40069f5a1d06b99c3ec5 => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:03:31.755819 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c6ad45-6e51-4d95-a38c-470a931dd4f0 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:31.756572 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c6ad45-6e51-4d95-a38c-470a931dd4f0 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:31.756572 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c6ad45-6e51-4d95-a38c-470a931dd4f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:31.780443 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-a6c6ad45-6e51-4d95-a38c-470a931dd4f0 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:03:31.781754 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-a6c6ad45-6e51-4d95-a38c-470a931dd4f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:31.866858 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1337/2660] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:03:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 01:03:31.877991 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66bf7ae7-77a4-4605-a85a-74bcc64d6059 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:31.879731 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-66bf7ae7-77a4-4605-a85a-74bcc64d6059 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:31.976540 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66bf7ae7-77a4-4605-a85a-74bcc64d6059 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:31.982545 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92441ae3-511f-4408-8364-37a1070e2a4b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:31.982545 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92441ae3-511f-4408-8364-37a1070e2a4b service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:31.982545 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92441ae3-511f-4408-8364-37a1070e2a4b service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:31.982545 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92441ae3-511f-4408-8364-37a1070e2a4b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:31.994007 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92441ae3-511f-4408-8364-37a1070e2a4b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:31.995258 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92441ae3-511f-4408-8364-37a1070e2a4b service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:32.009195 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1324/2661] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:03:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 132 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:03:32.443959 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c6ad45-6e51-4d95-a38c-470a931dd4f0 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:32.445904 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-a6c6ad45-6e51-4d95-a38c-470a931dd4f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:32.466097 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c6ad45-6e51-4d95-a38c-470a931dd4f0 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:32.472843 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c4e336-0282-4821-a923-966b425268df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:32.473497 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c4e336-0282-4821-a923-966b425268df admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:32.473788 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c4e336-0282-4821-a923-966b425268df admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:32.474064 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c4e336-0282-4821-a923-966b425268df admin admin] PATH_INFO: `/v3/projects/3097e5a686514e4ab558b5941f6efe43` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:32.475507 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43c4e336-0282-4821-a923-966b425268df admin admin] RBAC: Authorizing `identity:delete_project(project_id=3097e5a686514e4ab558b5941f6efe43)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:32.477789 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43c4e336-0282-4821-a923-966b425268df admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:32.557745 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-43c4e336-0282-4821-a923-966b425268df admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3097e5a686514e4ab558b5941f6efe43', 'request_id': 'req-43c4e336-0282-4821-a923-966b425268df'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:32.567657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-43c4e336-0282-4821-a923-966b425268df admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3097e5a686514e4ab558b5941f6efe43', 'request_id': 'req-43c4e336-0282-4821-a923-966b425268df'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:32.575912 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-43c4e336-0282-4821-a923-966b425268df admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3097e5a686514e4ab558b5941f6efe43', 'request_id': 'req-43c4e336-0282-4821-a923-966b425268df'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:32.590812 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-43c4e336-0282-4821-a923-966b425268df admin admin] The token cache is being invalidate because project 3097e5a686514e4ab558b5941f6efe43 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 01:03:32.591179 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-43c4e336-0282-4821-a923-966b425268df admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:32.591912 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-43c4e336-0282-4821-a923-966b425268df admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:32.593731 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1338/2662] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 01:03:32 2022] DELETE /identity/v3/projects/3097e5a686514e4ab558b5941f6efe43 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:03:32.598686 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92441ae3-511f-4408-8364-37a1070e2a4b service neutron] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:32.610109 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92441ae3-511f-4408-8364-37a1070e2a4b service neutron] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:32.697857 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0ae6d77-338c-4433-b2cb-649f52c1387f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:32.699086 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ae6d77-338c-4433-b2cb-649f52c1387f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:32.699609 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ae6d77-338c-4433-b2cb-649f52c1387f admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:32.700088 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ae6d77-338c-4433-b2cb-649f52c1387f admin admin] PATH_INFO: `/v3/users/d2aa47c8c2fd4164a4ffe5b9690e561b` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:32.702136 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0ae6d77-338c-4433-b2cb-649f52c1387f admin admin] RBAC: Authorizing `identity:delete_user(user_id=d2aa47c8c2fd4164a4ffe5b9690e561b)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:32.708435 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0ae6d77-338c-4433-b2cb-649f52c1387f admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:32.756889 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:32.779271 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-c0ae6d77-338c-4433-b2cb-649f52c1387f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd2aa47c8c2fd4164a4ffe5b9690e561b'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:32.791698 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-c0ae6d77-338c-4433-b2cb-649f52c1387f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd2aa47c8c2fd4164a4ffe5b9690e561b'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:32.792418 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-c0ae6d77-338c-4433-b2cb-649f52c1387f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd2aa47c8c2fd4164a4ffe5b9690e561b'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:32.802638 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-c0ae6d77-338c-4433-b2cb-649f52c1387f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd2aa47c8c2fd4164a4ffe5b9690e561b'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:32.803823 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-c0ae6d77-338c-4433-b2cb-649f52c1387f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd2aa47c8c2fd4164a4ffe5b9690e561b'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:32.835425 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1325/2663] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 01:03:32 2022] DELETE /identity/v3/users/d2aa47c8c2fd4164a4ffe5b9690e561b => generated 0 bytes in 238 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:03:33.178546 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c4e336-0282-4821-a923-966b425268df admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:33.180692 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-43c4e336-0282-4821-a923-966b425268df admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:33.196048 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c4e336-0282-4821-a923-966b425268df admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:33.290862 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e8b51c-2416-4394-aaa3-393ae0089600 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:33.291756 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e8b51c-2416-4394-aaa3-393ae0089600 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:33.292250 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e8b51c-2416-4394-aaa3-393ae0089600 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:33.292696 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e8b51c-2416-4394-aaa3-393ae0089600 admin admin] PATH_INFO: `/v3/projects/954dbc1f2fb246f98a56ee695b836bc2` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:33.294554 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01e8b51c-2416-4394-aaa3-393ae0089600 admin admin] RBAC: Authorizing `identity:delete_project(project_id=954dbc1f2fb246f98a56ee695b836bc2)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:33.296899 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01e8b51c-2416-4394-aaa3-393ae0089600 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:33.320978 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:33.381715 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-01e8b51c-2416-4394-aaa3-393ae0089600 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '954dbc1f2fb246f98a56ee695b836bc2', 'request_id': 'req-01e8b51c-2416-4394-aaa3-393ae0089600'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:33.395022 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-01e8b51c-2416-4394-aaa3-393ae0089600 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '954dbc1f2fb246f98a56ee695b836bc2', 'request_id': 'req-01e8b51c-2416-4394-aaa3-393ae0089600'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:33.405114 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0ae6d77-338c-4433-b2cb-649f52c1387f admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:33.405820 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-01e8b51c-2416-4394-aaa3-393ae0089600 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '954dbc1f2fb246f98a56ee695b836bc2', 'request_id': 'req-01e8b51c-2416-4394-aaa3-393ae0089600'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:33.407683 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-c0ae6d77-338c-4433-b2cb-649f52c1387f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:33.428298 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-01e8b51c-2416-4394-aaa3-393ae0089600 admin admin] The token cache is being invalidate because project 954dbc1f2fb246f98a56ee695b836bc2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 01:03:33.428770 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-01e8b51c-2416-4394-aaa3-393ae0089600 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:33.429854 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-01e8b51c-2416-4394-aaa3-393ae0089600 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:33.432244 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1339/2664] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 01:03:33 2022] DELETE /identity/v3/projects/954dbc1f2fb246f98a56ee695b836bc2 => generated 0 bytes in 255 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:03:33.439663 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0b63d5-26cc-40d2-be5d-3d7bbe7cff03 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:33.440124 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0b63d5-26cc-40d2-be5d-3d7bbe7cff03 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:33.440563 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0b63d5-26cc-40d2-be5d-3d7bbe7cff03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:33.473223 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-2f0b63d5-26cc-40d2-be5d-3d7bbe7cff03 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:03:33.474752 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-2f0b63d5-26cc-40d2-be5d-3d7bbe7cff03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:33.514508 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0ae6d77-338c-4433-b2cb-649f52c1387f admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:33.581228 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1340/2665] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:03:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 01:03:33.590571 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7949f04b-9953-4d1c-a202-d581175b16de None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:33.591301 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7949f04b-9953-4d1c-a202-d581175b16de None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:33.591960 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7949f04b-9953-4d1c-a202-d581175b16de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:33.595976 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59723ed5-f92b-49ea-9e17-3dad6c999ef8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:33.597560 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59723ed5-f92b-49ea-9e17-3dad6c999ef8 service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:33.598050 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59723ed5-f92b-49ea-9e17-3dad6c999ef8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:33.598529 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59723ed5-f92b-49ea-9e17-3dad6c999ef8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:33.601405 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-59723ed5-f92b-49ea-9e17-3dad6c999ef8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:33.617050 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-7949f04b-9953-4d1c-a202-d581175b16de None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:03:33.618673 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-7949f04b-9953-4d1c-a202-d581175b16de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:33.623453 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59723ed5-f92b-49ea-9e17-3dad6c999ef8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:33.625206 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59723ed5-f92b-49ea-9e17-3dad6c999ef8 service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:33.627777 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1341/2666] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:03:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 01:03:33.635348 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7949f04b-9953-4d1c-a202-d581175b16de None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:33.640720 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1326/2667] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:03:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 237 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 01:03:33.648786 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7949f04b-9953-4d1c-a202-d581175b16de None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:33.657145 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e548549e-691a-4ecd-ac66-c62c45ae0960 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:33.657977 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e548549e-691a-4ecd-ac66-c62c45ae0960 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:33.658326 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e548549e-691a-4ecd-ac66-c62c45ae0960 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:33.658630 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e548549e-691a-4ecd-ac66-c62c45ae0960 admin admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:33.659365 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e548549e-691a-4ecd-ac66-c62c45ae0960 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 01:03:33.660571 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e548549e-691a-4ecd-ac66-c62c45ae0960 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:33.662750 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e548549e-691a-4ecd-ac66-c62c45ae0960 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:33.678731 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1342/2668] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 01:03:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 01:03:33.683415 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59723ed5-f92b-49ea-9e17-3dad6c999ef8 service neutron] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:33.694818 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59723ed5-f92b-49ea-9e17-3dad6c999ef8 service neutron] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:33.700314 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8efd2952-3335-4faf-903f-156e71120832 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:33.701153 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efd2952-3335-4faf-903f-156e71120832 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:33.701599 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efd2952-3335-4faf-903f-156e71120832 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:33.702040 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efd2952-3335-4faf-903f-156e71120832 admin admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:33.703173 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8efd2952-3335-4faf-903f-156e71120832 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:33.706476 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8efd2952-3335-4faf-903f-156e71120832 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:33.745493 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1327/2669] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 01:03:33 2022] POST /identity/v3/projects => generated 387 bytes in 62 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 01:03:33.751458 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e548549e-691a-4ecd-ac66-c62c45ae0960 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:33.765403 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e548549e-691a-4ecd-ac66-c62c45ae0960 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:33.856849 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f356b9-3623-43e6-8769-dee712a048b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:33.858025 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f356b9-3623-43e6-8769-dee712a048b8 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:33.858510 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f356b9-3623-43e6-8769-dee712a048b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:33.859266 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f356b9-3623-43e6-8769-dee712a048b8 admin admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:33.860469 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1f356b9-3623-43e6-8769-dee712a048b8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:33.865219 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1f356b9-3623-43e6-8769-dee712a048b8 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:33.905576 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1343/2670] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 01:03:33 2022] POST /identity/v3/users => generated 294 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 01:03:33.912800 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8efd2952-3335-4faf-903f-156e71120832 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:33.926248 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8efd2952-3335-4faf-903f-156e71120832 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:33.931492 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77791956-b6af-49c6-b2b5-b2d32198c54e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:33.932374 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77791956-b6af-49c6-b2b5-b2d32198c54e admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:33.932814 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77791956-b6af-49c6-b2b5-b2d32198c54e admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:33.933247 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77791956-b6af-49c6-b2b5-b2d32198c54e admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:33.934121 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77791956-b6af-49c6-b2b5-b2d32198c54e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:33.936367 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77791956-b6af-49c6-b2b5-b2d32198c54e admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:33.951255 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1328/2671] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:03:33 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 01:03:33.956075 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f356b9-3623-43e6-8769-dee712a048b8 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:33.968375 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f356b9-3623-43e6-8769-dee712a048b8 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:33.974785 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d51f43c-92ca-4f44-9a5e-975a206c368f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:33.975407 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d51f43c-92ca-4f44-9a5e-975a206c368f admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:33.975682 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d51f43c-92ca-4f44-9a5e-975a206c368f admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:33.975951 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d51f43c-92ca-4f44-9a5e-975a206c368f admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:33.976612 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d51f43c-92ca-4f44-9a5e-975a206c368f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:33.978282 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d51f43c-92ca-4f44-9a5e-975a206c368f admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:33.991003 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1344/2672] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:03:33 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 01:03:33.996160 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77791956-b6af-49c6-b2b5-b2d32198c54e admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:34.013251 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77791956-b6af-49c6-b2b5-b2d32198c54e admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:34.021531 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d4299ad-cf17-4e96-a7c1-bef03919d5e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:34.022669 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4299ad-cf17-4e96-a7c1-bef03919d5e9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:34.023145 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4299ad-cf17-4e96-a7c1-bef03919d5e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:34.023601 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4299ad-cf17-4e96-a7c1-bef03919d5e9 admin admin] PATH_INFO: `/v3/projects/33eac46fae8d4b0ba481dbc87f4cc3c6/users/32e719144d704908a5ff164d216d7eda/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:34.053497 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d4299ad-cf17-4e96-a7c1-bef03919d5e9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=33eac46fae8d4b0ba481dbc87f4cc3c6, user_id=32e719144d704908a5ff164d216d7eda, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:34.059107 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d4299ad-cf17-4e96-a7c1-bef03919d5e9 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:34.084846 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1329/2673] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 01:03:33 2022] PUT /identity/v3/projects/33eac46fae8d4b0ba481dbc87f4cc3c6/users/32e719144d704908a5ff164d216d7eda/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 90 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:03:34.104820 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fa7606-d2a8-4ca9-80ee-eac66af8da11 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:34.105349 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fa7606-d2a8-4ca9-80ee-eac66af8da11 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:34.105827 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fa7606-d2a8-4ca9-80ee-eac66af8da11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:34.119961 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:34.159110 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-89fa7606-d2a8-4ca9-80ee-eac66af8da11 None None] MFA Rules not processed for user `32e719144d704908a5ff164d216d7eda`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:03:34.160640 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-89fa7606-d2a8-4ca9-80ee-eac66af8da11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:34.283562 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1345/2674] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:03:34 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:03:34.290735 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d4299ad-cf17-4e96-a7c1-bef03919d5e9 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:34.312244 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=92,keystone:DELETE=5,keystone:INSERT=3 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:34.413087 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d4299ad-cf17-4e96-a7c1-bef03919d5e9 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:34.421461 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc67cf4-bbad-4aa4-adfa-1f039ec26e9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:34.422504 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc67cf4-bbad-4aa4-adfa-1f039ec26e9d admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:34.422981 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc67cf4-bbad-4aa4-adfa-1f039ec26e9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:34.423450 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc67cf4-bbad-4aa4-adfa-1f039ec26e9d admin admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:34.424581 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcc67cf4-bbad-4aa4-adfa-1f039ec26e9d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:34.428910 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcc67cf4-bbad-4aa4-adfa-1f039ec26e9d admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:34.473784 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1330/2675] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 01:03:34 2022] POST /identity/v3/projects => generated 385 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 01:03:34.478954 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89fa7606-d2a8-4ca9-80ee-eac66af8da11 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:34.496998 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89fa7606-d2a8-4ca9-80ee-eac66af8da11 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:34.596554 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-934be3fe-0232-4073-a421-83a91685db1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:34.598055 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934be3fe-0232-4073-a421-83a91685db1a admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:34.598055 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934be3fe-0232-4073-a421-83a91685db1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:34.598055 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934be3fe-0232-4073-a421-83a91685db1a admin admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:34.598592 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-934be3fe-0232-4073-a421-83a91685db1a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:34.600636 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-934be3fe-0232-4073-a421-83a91685db1a admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:34.634671 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1346/2676] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 01:03:34 2022] POST /identity/v3/users => generated 299 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 01:03:34.640578 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc67cf4-bbad-4aa4-adfa-1f039ec26e9d admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:34.661453 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc67cf4-bbad-4aa4-adfa-1f039ec26e9d admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:34.671525 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c5d5b12-f223-4bdb-8350-f3671f789c27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:34.672585 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5d5b12-f223-4bdb-8350-f3671f789c27 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:34.673073 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5d5b12-f223-4bdb-8350-f3671f789c27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:34.673599 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5d5b12-f223-4bdb-8350-f3671f789c27 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:34.674801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c5d5b12-f223-4bdb-8350-f3671f789c27 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:34.678508 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c5d5b12-f223-4bdb-8350-f3671f789c27 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:34.695765 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1331/2677] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:03:34 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 01:03:34.702885 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-934be3fe-0232-4073-a421-83a91685db1a admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:34.722680 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-934be3fe-0232-4073-a421-83a91685db1a admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:34.731057 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bcffbe8-9bc4-4a4b-b3de-2fc9246d586f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:34.732243 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bcffbe8-9bc4-4a4b-b3de-2fc9246d586f admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:34.732738 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bcffbe8-9bc4-4a4b-b3de-2fc9246d586f admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:34.733229 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bcffbe8-9bc4-4a4b-b3de-2fc9246d586f admin admin] PATH_INFO: `/v3/projects/5c81c409e9dd480ba02daf4ca73f7a64/users/80557296b56c476cb18608bcff80cbc1/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:34.764727 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bcffbe8-9bc4-4a4b-b3de-2fc9246d586f admin admin] RBAC: Authorizing `identity:create_grant(project_id=5c81c409e9dd480ba02daf4ca73f7a64, user_id=80557296b56c476cb18608bcff80cbc1, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:34.770552 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bcffbe8-9bc4-4a4b-b3de-2fc9246d586f admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:34.796107 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1347/2678] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 01:03:34 2022] PUT /identity/v3/projects/5c81c409e9dd480ba02daf4ca73f7a64/users/80557296b56c476cb18608bcff80cbc1/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 94 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 01:03:34.802158 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c5d5b12-f223-4bdb-8350-f3671f789c27 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:34.821238 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c5d5b12-f223-4bdb-8350-f3671f789c27 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:34.920785 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d58122-a30e-4c77-9b6e-c4bd3f0acd48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:34.921828 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d58122-a30e-4c77-9b6e-c4bd3f0acd48 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:34.922287 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d58122-a30e-4c77-9b6e-c4bd3f0acd48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:34.922756 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d58122-a30e-4c77-9b6e-c4bd3f0acd48 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:34.923918 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d58122-a30e-4c77-9b6e-c4bd3f0acd48 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:34.927886 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d58122-a30e-4c77-9b6e-c4bd3f0acd48 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:34.955705 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1332/2679] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:03:34 2022] GET /identity/v3/roles => generated 1420 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 01:03:34.959856 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bcffbe8-9bc4-4a4b-b3de-2fc9246d586f admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:34.979831 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bcffbe8-9bc4-4a4b-b3de-2fc9246d586f admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:34.989650 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f21e034-f370-4dde-81ba-615bc2abf568 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:34.990722 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f21e034-f370-4dde-81ba-615bc2abf568 admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:34.991196 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f21e034-f370-4dde-81ba-615bc2abf568 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:34.992565 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f21e034-f370-4dde-81ba-615bc2abf568 admin admin] PATH_INFO: `/v3/projects/5c81c409e9dd480ba02daf4ca73f7a64/users/80557296b56c476cb18608bcff80cbc1/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:34.994657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f21e034-f370-4dde-81ba-615bc2abf568 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5c81c409e9dd480ba02daf4ca73f7a64, user_id=80557296b56c476cb18608bcff80cbc1, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:34.998014 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f21e034-f370-4dde-81ba-615bc2abf568 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:35.019280 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1348/2680] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 01:03:34 2022] PUT /identity/v3/projects/5c81c409e9dd480ba02daf4ca73f7a64/users/80557296b56c476cb18608bcff80cbc1/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:03:35.033509 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31cd4c80-967d-4b17-aa57-1b99e2659fea None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:35.034233 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31cd4c80-967d-4b17-aa57-1b99e2659fea None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:35.036647 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31cd4c80-967d-4b17-aa57-1b99e2659fea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:35.087487 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-31cd4c80-967d-4b17-aa57-1b99e2659fea None None] MFA Rules not processed for user `80557296b56c476cb18608bcff80cbc1`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:03:35.089060 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-31cd4c80-967d-4b17-aa57-1b99e2659fea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:35.218282 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1333/2681] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:03:35 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 01:03:35.228215 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f21e034-f370-4dde-81ba-615bc2abf568 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:35.230080 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-1f21e034-f370-4dde-81ba-615bc2abf568 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:35.495014 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:35.519443 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f21e034-f370-4dde-81ba-615bc2abf568 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:35.524536 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301ed53f-4189-4013-af63-7ef8c84073c1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:35.525179 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301ed53f-4189-4013-af63-7ef8c84073c1 service cinder] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:35.525487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301ed53f-4189-4013-af63-7ef8c84073c1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:35.525754 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301ed53f-4189-4013-af63-7ef8c84073c1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:35.543624 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-301ed53f-4189-4013-af63-7ef8c84073c1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:35.546548 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-301ed53f-4189-4013-af63-7ef8c84073c1 service cinder] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:35.566104 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1349/2682] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:03:35 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 01:03:35.617291 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31cd4c80-967d-4b17-aa57-1b99e2659fea None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:35.626548 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:35.728462 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31cd4c80-967d-4b17-aa57-1b99e2659fea None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:35.737645 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19bfe4e-21c3-48f8-9883-c53b882e5ac9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:35.738552 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19bfe4e-21c3-48f8-9883-c53b882e5ac9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:35.738902 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19bfe4e-21c3-48f8-9883-c53b882e5ac9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:35.739372 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19bfe4e-21c3-48f8-9883-c53b882e5ac9 admin admin] PATH_INFO: `/v3/users/32e719144d704908a5ff164d216d7eda` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:35.741547 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19bfe4e-21c3-48f8-9883-c53b882e5ac9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=32e719144d704908a5ff164d216d7eda)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:35.745244 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19bfe4e-21c3-48f8-9883-c53b882e5ac9 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:35.849100 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-c19bfe4e-21c3-48f8-9883-c53b882e5ac9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '32e719144d704908a5ff164d216d7eda'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:35.865482 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-c19bfe4e-21c3-48f8-9883-c53b882e5ac9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '32e719144d704908a5ff164d216d7eda'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:35.867040 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-c19bfe4e-21c3-48f8-9883-c53b882e5ac9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '32e719144d704908a5ff164d216d7eda'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:35.884121 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-c19bfe4e-21c3-48f8-9883-c53b882e5ac9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '32e719144d704908a5ff164d216d7eda'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:35.885422 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-c19bfe4e-21c3-48f8-9883-c53b882e5ac9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '32e719144d704908a5ff164d216d7eda'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:35.929993 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1334/2683] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 01:03:35 2022] DELETE /identity/v3/users/32e719144d704908a5ff164d216d7eda => generated 0 bytes in 312 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:03:35.934289 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7ae8ca-1845-4416-8ac3-a8e50bc86083 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:35.934885 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7ae8ca-1845-4416-8ac3-a8e50bc86083 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:35.935489 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7ae8ca-1845-4416-8ac3-a8e50bc86083 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:35.972674 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-0a7ae8ca-1845-4416-8ac3-a8e50bc86083 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:03:35.974670 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-0a7ae8ca-1845-4416-8ac3-a8e50bc86083 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:36.084227 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1350/2684] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:03:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:03:36.101416 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19bfe4e-21c3-48f8-9883-c53b882e5ac9 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:36.103349 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-c19bfe4e-21c3-48f8-9883-c53b882e5ac9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:36.201694 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19bfe4e-21c3-48f8-9883-c53b882e5ac9 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:36.205840 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610b6b47-70a1-44f8-baa1-990f6e179bcf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:36.206584 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610b6b47-70a1-44f8-baa1-990f6e179bcf service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:36.206870 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610b6b47-70a1-44f8-baa1-990f6e179bcf service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:36.207141 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610b6b47-70a1-44f8-baa1-990f6e179bcf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:36.219334 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-610b6b47-70a1-44f8-baa1-990f6e179bcf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:36.221536 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-610b6b47-70a1-44f8-baa1-990f6e179bcf service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:36.236104 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1335/2685] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:03:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 135 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 01:03:36.738641 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a7ae8ca-1845-4416-8ac3-a8e50bc86083 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:36.740886 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-0a7ae8ca-1845-4416-8ac3-a8e50bc86083 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:36.773022 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a7ae8ca-1845-4416-8ac3-a8e50bc86083 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:36.782424 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8fe3a92-9258-4e42-bb79-f90b101da89d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:36.783806 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8fe3a92-9258-4e42-bb79-f90b101da89d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:36.784235 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8fe3a92-9258-4e42-bb79-f90b101da89d admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:36.784735 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8fe3a92-9258-4e42-bb79-f90b101da89d admin admin] PATH_INFO: `/v3/projects/33eac46fae8d4b0ba481dbc87f4cc3c6` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:36.786919 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8fe3a92-9258-4e42-bb79-f90b101da89d admin admin] RBAC: Authorizing `identity:delete_project(project_id=33eac46fae8d4b0ba481dbc87f4cc3c6)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:36.790692 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8fe3a92-9258-4e42-bb79-f90b101da89d admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:36.868706 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-c8fe3a92-9258-4e42-bb79-f90b101da89d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '33eac46fae8d4b0ba481dbc87f4cc3c6', 'request_id': 'req-c8fe3a92-9258-4e42-bb79-f90b101da89d'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:36.880372 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-c8fe3a92-9258-4e42-bb79-f90b101da89d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '33eac46fae8d4b0ba481dbc87f4cc3c6', 'request_id': 'req-c8fe3a92-9258-4e42-bb79-f90b101da89d'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:36.894551 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-c8fe3a92-9258-4e42-bb79-f90b101da89d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '33eac46fae8d4b0ba481dbc87f4cc3c6', 'request_id': 'req-c8fe3a92-9258-4e42-bb79-f90b101da89d'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:36.916241 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-c8fe3a92-9258-4e42-bb79-f90b101da89d admin admin] The token cache is being invalidate because project 33eac46fae8d4b0ba481dbc87f4cc3c6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 01:03:36.916791 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-c8fe3a92-9258-4e42-bb79-f90b101da89d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:36.917828 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-c8fe3a92-9258-4e42-bb79-f90b101da89d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:36.919686 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1351/2686] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 01:03:36 2022] DELETE /identity/v3/projects/33eac46fae8d4b0ba481dbc87f4cc3c6 => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 01:03:36.926479 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610b6b47-70a1-44f8-baa1-990f6e179bcf service neutron] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:36.943635 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610b6b47-70a1-44f8-baa1-990f6e179bcf service neutron] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:36.993690 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:37.071329 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e4d49df-3400-4b9d-a68b-3f1882bd2aaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:37.072459 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4d49df-3400-4b9d-a68b-3f1882bd2aaa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:37.073072 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4d49df-3400-4b9d-a68b-3f1882bd2aaa admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:37.073531 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4d49df-3400-4b9d-a68b-3f1882bd2aaa admin admin] PATH_INFO: `/v3/users/80557296b56c476cb18608bcff80cbc1` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:37.075147 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e4d49df-3400-4b9d-a68b-3f1882bd2aaa admin admin] RBAC: Authorizing `identity:delete_user(user_id=80557296b56c476cb18608bcff80cbc1)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:37.078799 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e4d49df-3400-4b9d-a68b-3f1882bd2aaa admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:37.148510 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-6e4d49df-3400-4b9d-a68b-3f1882bd2aaa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '80557296b56c476cb18608bcff80cbc1'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:37.159674 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-6e4d49df-3400-4b9d-a68b-3f1882bd2aaa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '80557296b56c476cb18608bcff80cbc1'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:37.160780 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-6e4d49df-3400-4b9d-a68b-3f1882bd2aaa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '80557296b56c476cb18608bcff80cbc1'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:37.175827 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-6e4d49df-3400-4b9d-a68b-3f1882bd2aaa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '80557296b56c476cb18608bcff80cbc1'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:37.176915 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-6e4d49df-3400-4b9d-a68b-3f1882bd2aaa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '80557296b56c476cb18608bcff80cbc1'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:37.216307 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1336/2687] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 01:03:36 2022] DELETE /identity/v3/users/80557296b56c476cb18608bcff80cbc1 => generated 0 bytes in 292 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:03:37.739831 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8fe3a92-9258-4e42-bb79-f90b101da89d admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:37.742828 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-c8fe3a92-9258-4e42-bb79-f90b101da89d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:37.791523 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8fe3a92-9258-4e42-bb79-f90b101da89d admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:37.953336 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6aa1915-80d6-43f4-a4d7-8fb00a3a02a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:37.954441 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6aa1915-80d6-43f4-a4d7-8fb00a3a02a0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:37.954940 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6aa1915-80d6-43f4-a4d7-8fb00a3a02a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:37.955435 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6aa1915-80d6-43f4-a4d7-8fb00a3a02a0 admin admin] PATH_INFO: `/v3/projects/5c81c409e9dd480ba02daf4ca73f7a64` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:37.957820 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6aa1915-80d6-43f4-a4d7-8fb00a3a02a0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5c81c409e9dd480ba02daf4ca73f7a64)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:37.962145 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6aa1915-80d6-43f4-a4d7-8fb00a3a02a0 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:37.997049 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:38.079396 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-b6aa1915-80d6-43f4-a4d7-8fb00a3a02a0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5c81c409e9dd480ba02daf4ca73f7a64', 'request_id': 'req-b6aa1915-80d6-43f4-a4d7-8fb00a3a02a0'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:38.092786 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-b6aa1915-80d6-43f4-a4d7-8fb00a3a02a0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5c81c409e9dd480ba02daf4ca73f7a64', 'request_id': 'req-b6aa1915-80d6-43f4-a4d7-8fb00a3a02a0'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:38.105543 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-b6aa1915-80d6-43f4-a4d7-8fb00a3a02a0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5c81c409e9dd480ba02daf4ca73f7a64', 'request_id': 'req-b6aa1915-80d6-43f4-a4d7-8fb00a3a02a0'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:38.131122 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-b6aa1915-80d6-43f4-a4d7-8fb00a3a02a0 admin admin] The token cache is being invalidate because project 5c81c409e9dd480ba02daf4ca73f7a64 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 01:03:38.131884 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-b6aa1915-80d6-43f4-a4d7-8fb00a3a02a0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:38.133334 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-b6aa1915-80d6-43f4-a4d7-8fb00a3a02a0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:38.136515 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1352/2688] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 01:03:37 2022] DELETE /identity/v3/projects/5c81c409e9dd480ba02daf4ca73f7a64 => generated 0 bytes in 398 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:03:38.144775 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ca39da-3f03-4103-935b-771e2b4bdac7 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:38.145444 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ca39da-3f03-4103-935b-771e2b4bdac7 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:38.145782 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ca39da-3f03-4103-935b-771e2b4bdac7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:38.197890 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-88ca39da-3f03-4103-935b-771e2b4bdac7 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:03:38.199680 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-88ca39da-3f03-4103-935b-771e2b4bdac7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:38.328225 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1337/2689] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:03:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:03:38.336616 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee1725c-b9bb-4441-8277-9da67f3ed2ea None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:38.336911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee1725c-b9bb-4441-8277-9da67f3ed2ea None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:38.337188 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee1725c-b9bb-4441-8277-9da67f3ed2ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:38.366984 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-8ee1725c-b9bb-4441-8277-9da67f3ed2ea None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:03:38.371354 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-8ee1725c-b9bb-4441-8277-9da67f3ed2ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:38.403463 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1353/2690] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:03:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 68 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:03:38.406662 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ca39da-3f03-4103-935b-771e2b4bdac7 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:38.419739 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ca39da-3f03-4103-935b-771e2b4bdac7 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:38.425987 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245b5e5e-5903-469e-b507-61fc2a3ef2a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:38.426653 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245b5e5e-5903-469e-b507-61fc2a3ef2a1 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:38.427115 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245b5e5e-5903-469e-b507-61fc2a3ef2a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:38.427386 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245b5e5e-5903-469e-b507-61fc2a3ef2a1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:38.428078 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-245b5e5e-5903-469e-b507-61fc2a3ef2a1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 01:03:38.428461 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-245b5e5e-5903-469e-b507-61fc2a3ef2a1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:38.430222 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-245b5e5e-5903-469e-b507-61fc2a3ef2a1 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:38.449402 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1338/2691] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 01:03:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 01:03:38.457836 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee1725c-b9bb-4441-8277-9da67f3ed2ea None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:38.476094 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee1725c-b9bb-4441-8277-9da67f3ed2ea None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:38.487754 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87cf4a49-77af-4d5d-8ea1-22af91a080dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:38.489031 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87cf4a49-77af-4d5d-8ea1-22af91a080dd admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:38.489673 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87cf4a49-77af-4d5d-8ea1-22af91a080dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:38.490396 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87cf4a49-77af-4d5d-8ea1-22af91a080dd admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:38.491786 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87cf4a49-77af-4d5d-8ea1-22af91a080dd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:38.495261 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245b5e5e-5903-469e-b507-61fc2a3ef2a1 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:38.496060 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87cf4a49-77af-4d5d-8ea1-22af91a080dd admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:38.505229 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-245b5e5e-5903-469e-b507-61fc2a3ef2a1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:38.564644 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1354/2692] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 01:03:38 2022] POST /identity/v3/projects => generated 381 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 01:03:38.573975 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87cf4a49-77af-4d5d-8ea1-22af91a080dd admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:38.584792 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:38.652747 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245b5e5e-5903-469e-b507-61fc2a3ef2a1 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:38.694314 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87cf4a49-77af-4d5d-8ea1-22af91a080dd admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:38.700928 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e192821-53c6-4562-91f2-1020ce91f748 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:38.701838 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e192821-53c6-4562-91f2-1020ce91f748 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:38.702309 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e192821-53c6-4562-91f2-1020ce91f748 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:38.702771 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e192821-53c6-4562-91f2-1020ce91f748 admin admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:38.703663 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e192821-53c6-4562-91f2-1020ce91f748 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:38.706154 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e192821-53c6-4562-91f2-1020ce91f748 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:38.740320 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1355/2693] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 01:03:38 2022] POST /identity/v3/users => generated 291 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 01:03:38.741369 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e192821-53c6-4562-91f2-1020ce91f748 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:38.743194 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-0e192821-53c6-4562-91f2-1020ce91f748 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:38.744074 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38d213e-9df7-4c3b-990d-4382b1fd993c service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '11a80b55dcd444df901c8bf80f2a2c26', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:38.745443 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38d213e-9df7-4c3b-990d-4382b1fd993c service placement] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:38.746064 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38d213e-9df7-4c3b-990d-4382b1fd993c service placement] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:38.746540 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38d213e-9df7-4c3b-990d-4382b1fd993c service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:38.749538 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-b38d213e-9df7-4c3b-990d-4382b1fd993c service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:38.757703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e192821-53c6-4562-91f2-1020ce91f748 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:38.763772 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a689cc99-86d6-47a2-a9cb-f2eb7e4a1e5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:38.764856 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a689cc99-86d6-47a2-a9cb-f2eb7e4a1e5e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:38.765294 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a689cc99-86d6-47a2-a9cb-f2eb7e4a1e5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:38.765688 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a689cc99-86d6-47a2-a9cb-f2eb7e4a1e5e admin admin] PATH_INFO: `/v3/users/dbe551f8c10c4111b4ad7380a6482564` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:38.767193 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a689cc99-86d6-47a2-a9cb-f2eb7e4a1e5e admin admin] RBAC: Authorizing `identity:delete_user(user_id=dbe551f8c10c4111b4ad7380a6482564)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:38.770401 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a689cc99-86d6-47a2-a9cb-f2eb7e4a1e5e admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:38.772213 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b38d213e-9df7-4c3b-990d-4382b1fd993c service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:38.775189 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b38d213e-9df7-4c3b-990d-4382b1fd993c service placement] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:38.791867 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1339/2694] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:03:38 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 297 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 01:03:38.792747 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38d213e-9df7-4c3b-990d-4382b1fd993c service placement] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:38.804352 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38d213e-9df7-4c3b-990d-4382b1fd993c service placement] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:38.809779 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96c246fb-01ae-447b-93d8-a15b06ca54e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:38.810404 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c246fb-01ae-447b-93d8-a15b06ca54e8 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:38.810677 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c246fb-01ae-447b-93d8-a15b06ca54e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:38.810941 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c246fb-01ae-447b-93d8-a15b06ca54e8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:38.811706 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96c246fb-01ae-447b-93d8-a15b06ca54e8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:38.813600 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96c246fb-01ae-447b-93d8-a15b06ca54e8 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:38.838490 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1340/2695] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:03:38 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 01:03:38.845180 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96c246fb-01ae-447b-93d8-a15b06ca54e8 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:38.855648 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-a689cc99-86d6-47a2-a9cb-f2eb7e4a1e5e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dbe551f8c10c4111b4ad7380a6482564'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:38.856519 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-a689cc99-86d6-47a2-a9cb-f2eb7e4a1e5e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dbe551f8c10c4111b4ad7380a6482564'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:38.863937 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96c246fb-01ae-447b-93d8-a15b06ca54e8 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:38.869874 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-a689cc99-86d6-47a2-a9cb-f2eb7e4a1e5e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dbe551f8c10c4111b4ad7380a6482564'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:38.870920 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-a689cc99-86d6-47a2-a9cb-f2eb7e4a1e5e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dbe551f8c10c4111b4ad7380a6482564'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:38.871674 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a43d92-fbeb-46aa-85b0-caa26fae94a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:38.872857 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a43d92-fbeb-46aa-85b0-caa26fae94a2 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:38.873429 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a43d92-fbeb-46aa-85b0-caa26fae94a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:38.873928 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a43d92-fbeb-46aa-85b0-caa26fae94a2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:38.875215 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1a43d92-fbeb-46aa-85b0-caa26fae94a2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:38.875492 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=79,keystone:DELETE=15,keystone:INSERT=6 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:38.877945 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1a43d92-fbeb-46aa-85b0-caa26fae94a2 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:38.884223 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-a689cc99-86d6-47a2-a9cb-f2eb7e4a1e5e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dbe551f8c10c4111b4ad7380a6482564'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:38.894810 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1341/2696] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:03:38 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 01:03:38.900308 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a43d92-fbeb-46aa-85b0-caa26fae94a2 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:38.902470 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-a1a43d92-fbeb-46aa-85b0-caa26fae94a2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:38.912490 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1356/2697] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 01:03:38 2022] DELETE /identity/v3/users/dbe551f8c10c4111b4ad7380a6482564 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:03:38.919836 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a43d92-fbeb-46aa-85b0-caa26fae94a2 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:39.012256 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b934e71-f617-4af9-a183-2c26b759dcc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:39.013037 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b934e71-f617-4af9-a183-2c26b759dcc0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:39.013423 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b934e71-f617-4af9-a183-2c26b759dcc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:39.013845 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b934e71-f617-4af9-a183-2c26b759dcc0 admin admin] PATH_INFO: `/v3/projects/c6ce95ccfb2c41fb8255925082d67b4a/users/44734c71b752456683e25eaa97dda201/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:39.037873 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b934e71-f617-4af9-a183-2c26b759dcc0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c6ce95ccfb2c41fb8255925082d67b4a, user_id=44734c71b752456683e25eaa97dda201, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:39.040923 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b934e71-f617-4af9-a183-2c26b759dcc0 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:39.060538 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1342/2698] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 01:03:38 2022] PUT /identity/v3/projects/c6ce95ccfb2c41fb8255925082d67b4a/users/44734c71b752456683e25eaa97dda201/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 01:03:39.072413 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723dd21e-f257-439c-91f6-c68bb1467077 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:39.072688 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723dd21e-f257-439c-91f6-c68bb1467077 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:39.072952 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723dd21e-f257-439c-91f6-c68bb1467077 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:39.114348 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-723dd21e-f257-439c-91f6-c68bb1467077 None None] MFA Rules not processed for user `44734c71b752456683e25eaa97dda201`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:03:39.116724 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-723dd21e-f257-439c-91f6-c68bb1467077 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:39.195949 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b934e71-f617-4af9-a183-2c26b759dcc0 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:39.199982 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-5b934e71-f617-4af9-a183-2c26b759dcc0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:39.245651 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1357/2699] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:03:39 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 01:03:39.250998 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723dd21e-f257-439c-91f6-c68bb1467077 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:39.269234 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:39.320714 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b934e71-f617-4af9-a183-2c26b759dcc0 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:39.329996 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a4c94d3-dd65-48b7-9d0b-ad4dbc539f15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:39.330823 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4c94d3-dd65-48b7-9d0b-ad4dbc539f15 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:39.331825 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4c94d3-dd65-48b7-9d0b-ad4dbc539f15 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:39.332179 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4c94d3-dd65-48b7-9d0b-ad4dbc539f15 admin admin] PATH_INFO: `/v3/projects/f09655506e354c9d991f4df9a92f4086` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:39.333774 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a4c94d3-dd65-48b7-9d0b-ad4dbc539f15 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f09655506e354c9d991f4df9a92f4086)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:39.335956 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a4c94d3-dd65-48b7-9d0b-ad4dbc539f15 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:39.337397 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723dd21e-f257-439c-91f6-c68bb1467077 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:39.343442 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7452f89b-2970-496f-a2d6-489f9835150c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:39.344226 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7452f89b-2970-496f-a2d6-489f9835150c admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:39.344563 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7452f89b-2970-496f-a2d6-489f9835150c admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:39.344956 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7452f89b-2970-496f-a2d6-489f9835150c admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:39.345780 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7452f89b-2970-496f-a2d6-489f9835150c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:39.348064 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7452f89b-2970-496f-a2d6-489f9835150c admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:39.399305 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1358/2700] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 01:03:39 2022] POST /identity/v3/projects => generated 381 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 01:03:39.407227 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7452f89b-2970-496f-a2d6-489f9835150c admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:39.410341 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-0a4c94d3-dd65-48b7-9d0b-ad4dbc539f15 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f09655506e354c9d991f4df9a92f4086', 'request_id': 'req-0a4c94d3-dd65-48b7-9d0b-ad4dbc539f15'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:39.421107 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-0a4c94d3-dd65-48b7-9d0b-ad4dbc539f15 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f09655506e354c9d991f4df9a92f4086', 'request_id': 'req-0a4c94d3-dd65-48b7-9d0b-ad4dbc539f15'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:39.432358 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-0a4c94d3-dd65-48b7-9d0b-ad4dbc539f15 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f09655506e354c9d991f4df9a92f4086', 'request_id': 'req-0a4c94d3-dd65-48b7-9d0b-ad4dbc539f15'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:39.453616 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-0a4c94d3-dd65-48b7-9d0b-ad4dbc539f15 admin admin] The token cache is being invalidate because project f09655506e354c9d991f4df9a92f4086 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81386) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 01:03:39.453986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-0a4c94d3-dd65-48b7-9d0b-ad4dbc539f15 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:39.455029 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-0a4c94d3-dd65-48b7-9d0b-ad4dbc539f15 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:39.457647 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1343/2701] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 01:03:39 2022] DELETE /identity/v3/projects/f09655506e354c9d991f4df9a92f4086 => generated 0 bytes in 263 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 01:03:39.465037 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eedc93e1-78fb-475b-9d17-5ea65d1cb9ee None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:39.465544 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eedc93e1-78fb-475b-9d17-5ea65d1cb9ee None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:39.465960 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eedc93e1-78fb-475b-9d17-5ea65d1cb9ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:39.496493 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-eedc93e1-78fb-475b-9d17-5ea65d1cb9ee None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:03:39.499020 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-eedc93e1-78fb-475b-9d17-5ea65d1cb9ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:39.514383 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=97,keystone:DELETE=2,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:39.532993 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7452f89b-2970-496f-a2d6-489f9835150c admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:39.540395 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e24a0c14-dfc7-488e-bb02-0aed02ab2828 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:39.541029 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24a0c14-dfc7-488e-bb02-0aed02ab2828 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:39.541370 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24a0c14-dfc7-488e-bb02-0aed02ab2828 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:39.541653 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24a0c14-dfc7-488e-bb02-0aed02ab2828 admin admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:39.542301 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e24a0c14-dfc7-488e-bb02-0aed02ab2828 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:39.544285 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e24a0c14-dfc7-488e-bb02-0aed02ab2828 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:39.574886 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1359/2702] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 01:03:39 2022] POST /identity/v3/users => generated 297 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 01:03:39.582185 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e24a0c14-dfc7-488e-bb02-0aed02ab2828 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:39.584254 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-e24a0c14-dfc7-488e-bb02-0aed02ab2828 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:39.602577 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e24a0c14-dfc7-488e-bb02-0aed02ab2828 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:39.607485 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf4182f-2224-4638-84ee-4f9c8fd35450 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:39.607984 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1344/2703] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:03:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 01:03:39.608187 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf4182f-2224-4638-84ee-4f9c8fd35450 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:39.608538 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf4182f-2224-4638-84ee-4f9c8fd35450 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:39.608817 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf4182f-2224-4638-84ee-4f9c8fd35450 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:39.609771 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf4182f-2224-4638-84ee-4f9c8fd35450 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:39.611393 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf4182f-2224-4638-84ee-4f9c8fd35450 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:39.618019 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7278da2-bb7b-4152-96b0-d39968151346 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:39.618597 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7278da2-bb7b-4152-96b0-d39968151346 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:39.619075 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7278da2-bb7b-4152-96b0-d39968151346 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:39.623342 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1360/2704] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:03:39 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 01:03:39.629837 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf4182f-2224-4638-84ee-4f9c8fd35450 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:39.646150 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf4182f-2224-4638-84ee-4f9c8fd35450 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:39.653029 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b8ffaf-ffb3-4f4d-b9f8-3671962dcc23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:39.654131 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b8ffaf-ffb3-4f4d-b9f8-3671962dcc23 admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:39.654510 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b8ffaf-ffb3-4f4d-b9f8-3671962dcc23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:39.654923 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-c7278da2-bb7b-4152-96b0-d39968151346 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:03:39.655108 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b8ffaf-ffb3-4f4d-b9f8-3671962dcc23 admin admin] PATH_INFO: `/v3/projects/4b951ae6cc5843f2b6af8555d739bdf2/users/9fef266826ea4f55a4dee661b086e363/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:39.656309 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-c7278da2-bb7b-4152-96b0-d39968151346 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:39.666613 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1345/2705] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:03:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 01:03:39.671323 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7278da2-bb7b-4152-96b0-d39968151346 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:39.678262 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08b8ffaf-ffb3-4f4d-b9f8-3671962dcc23 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4b951ae6cc5843f2b6af8555d739bdf2, user_id=9fef266826ea4f55a4dee661b086e363, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:39.680945 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08b8ffaf-ffb3-4f4d-b9f8-3671962dcc23 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:39.684838 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7278da2-bb7b-4152-96b0-d39968151346 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:39.693452 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-916ac5bf-e506-4697-8164-ec6c3a783982 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:39.694058 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-916ac5bf-e506-4697-8164-ec6c3a783982 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:39.694329 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-916ac5bf-e506-4697-8164-ec6c3a783982 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:39.694589 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-916ac5bf-e506-4697-8164-ec6c3a783982 admin admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:39.695235 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-916ac5bf-e506-4697-8164-ec6c3a783982 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 01:03:39.695531 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-916ac5bf-e506-4697-8164-ec6c3a783982 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:39.697131 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-916ac5bf-e506-4697-8164-ec6c3a783982 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:39.701215 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1361/2706] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 01:03:39 2022] PUT /identity/v3/projects/4b951ae6cc5843f2b6af8555d739bdf2/users/9fef266826ea4f55a4dee661b086e363/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 73 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 01:03:39.705759 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b8ffaf-ffb3-4f4d-b9f8-3671962dcc23 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:39.720835 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1346/2707] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 01:03:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 01:03:39.729329 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-916ac5bf-e506-4697-8164-ec6c3a783982 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:39.741885 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-916ac5bf-e506-4697-8164-ec6c3a783982 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:39.786998 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b8ffaf-ffb3-4f4d-b9f8-3671962dcc23 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:39.792918 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1df38601-9ab4-423c-bd66-f5fe335c80ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:39.793729 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df38601-9ab4-423c-bd66-f5fe335c80ef admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:39.794110 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df38601-9ab4-423c-bd66-f5fe335c80ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:39.794555 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df38601-9ab4-423c-bd66-f5fe335c80ef admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:39.795470 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1df38601-9ab4-423c-bd66-f5fe335c80ef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:39.797404 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1df38601-9ab4-423c-bd66-f5fe335c80ef admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:39.807969 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:39.808996 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1362/2708] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:03:39 2022] GET /identity/v3/roles => generated 1420 bytes in 104 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 01:03:39.813476 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1df38601-9ab4-423c-bd66-f5fe335c80ef admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:39.825044 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1df38601-9ab4-423c-bd66-f5fe335c80ef admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:39.833319 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27f44c56-7f13-45fb-84c9-bf6e5c72fdca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:39.834001 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f44c56-7f13-45fb-84c9-bf6e5c72fdca admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:39.834268 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f44c56-7f13-45fb-84c9-bf6e5c72fdca admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:39.834545 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f44c56-7f13-45fb-84c9-bf6e5c72fdca admin admin] PATH_INFO: `/v3/projects/4b951ae6cc5843f2b6af8555d739bdf2/users/9fef266826ea4f55a4dee661b086e363/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:39.837381 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27f44c56-7f13-45fb-84c9-bf6e5c72fdca admin admin] RBAC: Authorizing `identity:create_grant(project_id=4b951ae6cc5843f2b6af8555d739bdf2, user_id=9fef266826ea4f55a4dee661b086e363, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:39.840327 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27f44c56-7f13-45fb-84c9-bf6e5c72fdca admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:39.853681 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80363fd5-e401-4021-a62e-ecc524ae54ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:39.854692 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80363fd5-e401-4021-a62e-ecc524ae54ed admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:39.855000 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1363/2709] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 01:03:39 2022] PUT /identity/v3/projects/4b951ae6cc5843f2b6af8555d739bdf2/users/9fef266826ea4f55a4dee661b086e363/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 42 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:03:39.855189 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80363fd5-e401-4021-a62e-ecc524ae54ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:39.855617 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80363fd5-e401-4021-a62e-ecc524ae54ed admin admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:39.856775 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80363fd5-e401-4021-a62e-ecc524ae54ed admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:39.860686 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80363fd5-e401-4021-a62e-ecc524ae54ed admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:39.866830 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afc9a46-a5c4-41b7-801b-d5a2d288ebd2 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:39.867105 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afc9a46-a5c4-41b7-801b-d5a2d288ebd2 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:39.867365 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afc9a46-a5c4-41b7-801b-d5a2d288ebd2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:39.902641 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-0afc9a46-a5c4-41b7-801b-d5a2d288ebd2 None None] MFA Rules not processed for user `9fef266826ea4f55a4dee661b086e363`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:03:39.903874 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-0afc9a46-a5c4-41b7-801b-d5a2d288ebd2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:39.913374 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1347/2710] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 01:03:39 2022] POST /identity/v3/projects => generated 375 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 01:03:39.914453 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80363fd5-e401-4021-a62e-ecc524ae54ed admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:40.010130 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80363fd5-e401-4021-a62e-ecc524ae54ed admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:40.015104 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1364/2711] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:03:39 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 01:03:40.016584 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e98721a-e94e-4eab-8307-bd83f579ac95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:40.017422 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e98721a-e94e-4eab-8307-bd83f579ac95 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:40.017786 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e98721a-e94e-4eab-8307-bd83f579ac95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:40.018155 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e98721a-e94e-4eab-8307-bd83f579ac95 admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:40.019706 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e98721a-e94e-4eab-8307-bd83f579ac95 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:40.022199 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e98721a-e94e-4eab-8307-bd83f579ac95 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:40.030851 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0afc9a46-a5c4-41b7-801b-d5a2d288ebd2 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:40.032479 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-0afc9a46-a5c4-41b7-801b-d5a2d288ebd2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:40.049426 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:40.054727 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1348/2712] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 01:03:39 2022] POST /identity/v3/users => generated 288 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 01:03:40.060263 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e98721a-e94e-4eab-8307-bd83f579ac95 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:40.079333 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e98721a-e94e-4eab-8307-bd83f579ac95 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:40.086801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c85dc49f-15c7-4c15-8944-8c94f8a88722 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:40.087416 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85dc49f-15c7-4c15-8944-8c94f8a88722 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:40.087688 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85dc49f-15c7-4c15-8944-8c94f8a88722 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:40.088041 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85dc49f-15c7-4c15-8944-8c94f8a88722 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:40.088701 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c85dc49f-15c7-4c15-8944-8c94f8a88722 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:40.090365 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c85dc49f-15c7-4c15-8944-8c94f8a88722 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:40.102787 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1349/2713] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:03:40 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 01:03:40.107980 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c85dc49f-15c7-4c15-8944-8c94f8a88722 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:40.121072 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c85dc49f-15c7-4c15-8944-8c94f8a88722 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:40.121908 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0afc9a46-a5c4-41b7-801b-d5a2d288ebd2 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:40.126151 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce69a75-13fc-4050-a867-dddc972dd5d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:40.126781 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce69a75-13fc-4050-a867-dddc972dd5d4 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:40.127054 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce69a75-13fc-4050-a867-dddc972dd5d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:40.127311 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce69a75-13fc-4050-a867-dddc972dd5d4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:40.128026 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ce69a75-13fc-4050-a867-dddc972dd5d4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:40.129721 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ce69a75-13fc-4050-a867-dddc972dd5d4 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:40.141170 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1350/2714] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:03:40 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 01:03:40.146553 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce69a75-13fc-4050-a867-dddc972dd5d4 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:40.159603 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce69a75-13fc-4050-a867-dddc972dd5d4 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:40.165119 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-331ab1b0-78b6-4e27-ad5c-c71ca87a4753 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:40.165960 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331ab1b0-78b6-4e27-ad5c-c71ca87a4753 admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:40.166233 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331ab1b0-78b6-4e27-ad5c-c71ca87a4753 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:40.166501 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331ab1b0-78b6-4e27-ad5c-c71ca87a4753 admin admin] PATH_INFO: `/v3/projects/f37b53a21b19482b8dcac6c38505a50b/users/1e515c9c8dcf4a1aa7dd74fb7655adf4/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:40.190715 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-331ab1b0-78b6-4e27-ad5c-c71ca87a4753 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f37b53a21b19482b8dcac6c38505a50b, user_id=1e515c9c8dcf4a1aa7dd74fb7655adf4, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:40.194949 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-331ab1b0-78b6-4e27-ad5c-c71ca87a4753 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:40.208111 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6db1934f-21d0-44f1-bbbb-9859c2f58e96 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:40.209170 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db1934f-21d0-44f1-bbbb-9859c2f58e96 service cinder] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:40.209870 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db1934f-21d0-44f1-bbbb-9859c2f58e96 service cinder] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:40.210306 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db1934f-21d0-44f1-bbbb-9859c2f58e96 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:40.218525 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1351/2715] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 01:03:40 2022] PUT /identity/v3/projects/f37b53a21b19482b8dcac6c38505a50b/users/1e515c9c8dcf4a1aa7dd74fb7655adf4/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 72 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 01:03:40.224396 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03dfe948-6fde-44ee-aee4-a01b3c01eb3a None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:40.224742 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03dfe948-6fde-44ee-aee4-a01b3c01eb3a None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:40.225152 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03dfe948-6fde-44ee-aee4-a01b3c01eb3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:40.251675 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-03dfe948-6fde-44ee-aee4-a01b3c01eb3a None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:03:40.255301 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-03dfe948-6fde-44ee-aee4-a01b3c01eb3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:40.292376 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6db1934f-21d0-44f1-bbbb-9859c2f58e96 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:40.293862 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6db1934f-21d0-44f1-bbbb-9859c2f58e96 service cinder] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:40.309689 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1365/2716] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:03:40 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 279 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:03:40.341410 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1352/2717] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:03:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 01:03:40.359308 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6db1934f-21d0-44f1-bbbb-9859c2f58e96 service cinder] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:40.364677 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-6db1934f-21d0-44f1-bbbb-9859c2f58e96 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:40.463897 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6db1934f-21d0-44f1-bbbb-9859c2f58e96 service cinder] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:40.521113 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:40.549763 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22622829-4cad-435d-932c-28afc31ccc59 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:40.551046 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22622829-4cad-435d-932c-28afc31ccc59 service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:40.551046 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22622829-4cad-435d-932c-28afc31ccc59 service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:40.551046 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22622829-4cad-435d-932c-28afc31ccc59 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:40.561769 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22622829-4cad-435d-932c-28afc31ccc59 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:40.563267 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22622829-4cad-435d-932c-28afc31ccc59 service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:40.574918 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1366/2718] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:03:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 217 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:03:40.788859 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03dfe948-6fde-44ee-aee4-a01b3c01eb3a None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:40.804762 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:40.890908 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03dfe948-6fde-44ee-aee4-a01b3c01eb3a None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:40.899937 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9d5597-a095-419d-aaaf-6192f47eba02 tempest-VolumeQuotaClassesTest-1833465535 tempest-VolumeQuotaClassesTest-1833465535-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9fef266826ea4f55a4dee661b086e363', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4b951ae6cc5843f2b6af8555d739bdf2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:40.900691 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9d5597-a095-419d-aaaf-6192f47eba02 tempest-VolumeQuotaClassesTest-1833465535 tempest-VolumeQuotaClassesTest-1833465535-project-admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:40.901011 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9d5597-a095-419d-aaaf-6192f47eba02 tempest-VolumeQuotaClassesTest-1833465535 tempest-VolumeQuotaClassesTest-1833465535-project-admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:40.901438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9d5597-a095-419d-aaaf-6192f47eba02 tempest-VolumeQuotaClassesTest-1833465535 tempest-VolumeQuotaClassesTest-1833465535-project-admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:40.902245 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c9d5597-a095-419d-aaaf-6192f47eba02 tempest-VolumeQuotaClassesTest-1833465535 tempest-VolumeQuotaClassesTest-1833465535-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 01:03:40.902650 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c9d5597-a095-419d-aaaf-6192f47eba02 tempest-VolumeQuotaClassesTest-1833465535 tempest-VolumeQuotaClassesTest-1833465535-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:40.904604 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c9d5597-a095-419d-aaaf-6192f47eba02 tempest-VolumeQuotaClassesTest-1833465535 tempest-VolumeQuotaClassesTest-1833465535-project-admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:40.922244 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1353/2719] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 01:03:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 01:03:40.927689 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22622829-4cad-435d-932c-28afc31ccc59 service neutron] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:40.939007 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22622829-4cad-435d-932c-28afc31ccc59 service neutron] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:40.945429 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8acf919c-d8fb-44e8-8e7c-c0541f8b3895 tempest-VolumeQuotaClassesTest-1833465535 tempest-VolumeQuotaClassesTest-1833465535-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9fef266826ea4f55a4dee661b086e363', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4b951ae6cc5843f2b6af8555d739bdf2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:40.946155 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8acf919c-d8fb-44e8-8e7c-c0541f8b3895 tempest-VolumeQuotaClassesTest-1833465535 tempest-VolumeQuotaClassesTest-1833465535-project-admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:40.946501 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8acf919c-d8fb-44e8-8e7c-c0541f8b3895 tempest-VolumeQuotaClassesTest-1833465535 tempest-VolumeQuotaClassesTest-1833465535-project-admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:40.946841 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8acf919c-d8fb-44e8-8e7c-c0541f8b3895 tempest-VolumeQuotaClassesTest-1833465535 tempest-VolumeQuotaClassesTest-1833465535-project-admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:40.947566 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8acf919c-d8fb-44e8-8e7c-c0541f8b3895 tempest-VolumeQuotaClassesTest-1833465535 tempest-VolumeQuotaClassesTest-1833465535-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:40.949700 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8acf919c-d8fb-44e8-8e7c-c0541f8b3895 tempest-VolumeQuotaClassesTest-1833465535 tempest-VolumeQuotaClassesTest-1833465535-project-admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:40.997582 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1367/2720] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 01:03:40 2022] POST /identity/v3/projects => generated 354 bytes in 70 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 01:03:41.002522 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9d5597-a095-419d-aaaf-6192f47eba02 tempest-VolumeQuotaClassesTest-1833465535 tempest-VolumeQuotaClassesTest-1833465535-project-admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:41.021496 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9d5597-a095-419d-aaaf-6192f47eba02 tempest-VolumeQuotaClassesTest-1833465535 tempest-VolumeQuotaClassesTest-1833465535-project-admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:41.140549 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a09af212-576e-409e-8128-cf188b02d773 tempest-VolumeQuotaClassesTest-1833465535 tempest-VolumeQuotaClassesTest-1833465535-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9fef266826ea4f55a4dee661b086e363', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4b951ae6cc5843f2b6af8555d739bdf2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:41.141897 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09af212-576e-409e-8128-cf188b02d773 tempest-VolumeQuotaClassesTest-1833465535 tempest-VolumeQuotaClassesTest-1833465535-project-admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:41.142490 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09af212-576e-409e-8128-cf188b02d773 tempest-VolumeQuotaClassesTest-1833465535 tempest-VolumeQuotaClassesTest-1833465535-project-admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:41.143087 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09af212-576e-409e-8128-cf188b02d773 tempest-VolumeQuotaClassesTest-1833465535 tempest-VolumeQuotaClassesTest-1833465535-project-admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:41.145627 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a09af212-576e-409e-8128-cf188b02d773 tempest-VolumeQuotaClassesTest-1833465535 tempest-VolumeQuotaClassesTest-1833465535-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 01:03:41.146337 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a09af212-576e-409e-8128-cf188b02d773 tempest-VolumeQuotaClassesTest-1833465535 tempest-VolumeQuotaClassesTest-1833465535-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:41.149743 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a09af212-576e-409e-8128-cf188b02d773 tempest-VolumeQuotaClassesTest-1833465535 tempest-VolumeQuotaClassesTest-1833465535-project-admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:41.186701 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1354/2721] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 01:03:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 01:03:41.248441 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8acf919c-d8fb-44e8-8e7c-c0541f8b3895 tempest-VolumeQuotaClassesTest-1833465535 tempest-VolumeQuotaClassesTest-1833465535-project-admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:41.264202 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8acf919c-d8fb-44e8-8e7c-c0541f8b3895 tempest-VolumeQuotaClassesTest-1833465535 tempest-VolumeQuotaClassesTest-1833465535-project-admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:41.270207 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95f54f8a-9e47-4f9d-ab54-6692115532b8 tempest-VolumeQuotaClassesTest-1833465535 tempest-VolumeQuotaClassesTest-1833465535-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9fef266826ea4f55a4dee661b086e363', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4b951ae6cc5843f2b6af8555d739bdf2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:41.271129 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f54f8a-9e47-4f9d-ab54-6692115532b8 tempest-VolumeQuotaClassesTest-1833465535 tempest-VolumeQuotaClassesTest-1833465535-project-admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:41.271454 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f54f8a-9e47-4f9d-ab54-6692115532b8 tempest-VolumeQuotaClassesTest-1833465535 tempest-VolumeQuotaClassesTest-1833465535-project-admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:41.271773 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f54f8a-9e47-4f9d-ab54-6692115532b8 tempest-VolumeQuotaClassesTest-1833465535 tempest-VolumeQuotaClassesTest-1833465535-project-admin] PATH_INFO: `/v3/projects/2777b0fe5317464c86f911db05498020` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:41.273464 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95f54f8a-9e47-4f9d-ab54-6692115532b8 tempest-VolumeQuotaClassesTest-1833465535 tempest-VolumeQuotaClassesTest-1833465535-project-admin] RBAC: Authorizing `identity:delete_project(project_id=2777b0fe5317464c86f911db05498020)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:41.275714 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95f54f8a-9e47-4f9d-ab54-6692115532b8 tempest-VolumeQuotaClassesTest-1833465535 tempest-VolumeQuotaClassesTest-1833465535-project-admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:41.360869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-95f54f8a-9e47-4f9d-ab54-6692115532b8 tempest-VolumeQuotaClassesTest-1833465535 tempest-VolumeQuotaClassesTest-1833465535-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2777b0fe5317464c86f911db05498020', 'request_id': 'req-95f54f8a-9e47-4f9d-ab54-6692115532b8'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:41.372795 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-95f54f8a-9e47-4f9d-ab54-6692115532b8 tempest-VolumeQuotaClassesTest-1833465535 tempest-VolumeQuotaClassesTest-1833465535-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2777b0fe5317464c86f911db05498020', 'request_id': 'req-95f54f8a-9e47-4f9d-ab54-6692115532b8'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:41.384653 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-95f54f8a-9e47-4f9d-ab54-6692115532b8 tempest-VolumeQuotaClassesTest-1833465535 tempest-VolumeQuotaClassesTest-1833465535-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2777b0fe5317464c86f911db05498020', 'request_id': 'req-95f54f8a-9e47-4f9d-ab54-6692115532b8'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:41.407486 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-95f54f8a-9e47-4f9d-ab54-6692115532b8 tempest-VolumeQuotaClassesTest-1833465535 tempest-VolumeQuotaClassesTest-1833465535-project-admin] The token cache is being invalidate because project 2777b0fe5317464c86f911db05498020 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 01:03:41.408175 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-95f54f8a-9e47-4f9d-ab54-6692115532b8 tempest-VolumeQuotaClassesTest-1833465535 tempest-VolumeQuotaClassesTest-1833465535-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:41.409605 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-95f54f8a-9e47-4f9d-ab54-6692115532b8 tempest-VolumeQuotaClassesTest-1833465535 tempest-VolumeQuotaClassesTest-1833465535-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:41.412537 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1368/2722] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 01:03:41 2022] DELETE /identity/v3/projects/2777b0fe5317464c86f911db05498020 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:03:41.801208 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a09af212-576e-409e-8128-cf188b02d773 tempest-VolumeQuotaClassesTest-1833465535 tempest-VolumeQuotaClassesTest-1833465535-project-admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:41.803845 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-a09af212-576e-409e-8128-cf188b02d773 tempest-VolumeQuotaClassesTest-1833465535 tempest-VolumeQuotaClassesTest-1833465535-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:41.898084 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a09af212-576e-409e-8128-cf188b02d773 tempest-VolumeQuotaClassesTest-1833465535 tempest-VolumeQuotaClassesTest-1833465535-project-admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:41.955691 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:41.981165 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e09405-fd94-40f9-9369-4a9b6ed765db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:41.982157 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e09405-fd94-40f9-9369-4a9b6ed765db admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:41.982745 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e09405-fd94-40f9-9369-4a9b6ed765db admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:41.984523 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e09405-fd94-40f9-9369-4a9b6ed765db admin admin] PATH_INFO: `/v3/users/44734c71b752456683e25eaa97dda201` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:41.985328 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e09405-fd94-40f9-9369-4a9b6ed765db admin admin] RBAC: Authorizing `identity:delete_user(user_id=44734c71b752456683e25eaa97dda201)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:41.987830 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e09405-fd94-40f9-9369-4a9b6ed765db admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:42.082931 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-97e09405-fd94-40f9-9369-4a9b6ed765db admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '44734c71b752456683e25eaa97dda201'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:42.100451 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-97e09405-fd94-40f9-9369-4a9b6ed765db admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '44734c71b752456683e25eaa97dda201'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:42.101702 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-97e09405-fd94-40f9-9369-4a9b6ed765db admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '44734c71b752456683e25eaa97dda201'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:42.120500 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-97e09405-fd94-40f9-9369-4a9b6ed765db admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '44734c71b752456683e25eaa97dda201'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:42.121800 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-97e09405-fd94-40f9-9369-4a9b6ed765db admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '44734c71b752456683e25eaa97dda201'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:42.172595 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1355/2723] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 01:03:41 2022] DELETE /identity/v3/users/44734c71b752456683e25eaa97dda201 => generated 0 bytes in 368 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:03:42.176263 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5426df3c-f6d9-4805-870f-92a43233121d None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:42.177303 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5426df3c-f6d9-4805-870f-92a43233121d None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:42.177644 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5426df3c-f6d9-4805-870f-92a43233121d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:42.200461 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-5426df3c-f6d9-4805-870f-92a43233121d None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:03:42.202035 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-5426df3c-f6d9-4805-870f-92a43233121d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:42.337907 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1369/2724] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:03:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:03:42.354483 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e09405-fd94-40f9-9369-4a9b6ed765db admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:42.360613 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-97e09405-fd94-40f9-9369-4a9b6ed765db admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:42.463990 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e09405-fd94-40f9-9369-4a9b6ed765db admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:42.477421 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614f4f7b-b6ab-408b-9a86-9077ccffd17e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:42.478710 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614f4f7b-b6ab-408b-9a86-9077ccffd17e service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:42.479275 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614f4f7b-b6ab-408b-9a86-9077ccffd17e service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:42.479821 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614f4f7b-b6ab-408b-9a86-9077ccffd17e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:42.499519 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-614f4f7b-b6ab-408b-9a86-9077ccffd17e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:42.502372 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-614f4f7b-b6ab-408b-9a86-9077ccffd17e service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:42.523953 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1356/2725] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:03:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:03:43.261692 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5426df3c-f6d9-4805-870f-92a43233121d None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:43.263521 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-5426df3c-f6d9-4805-870f-92a43233121d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:43.279261 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5426df3c-f6d9-4805-870f-92a43233121d None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:43.289971 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eee3459-78be-4b93-9404-65865183177d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:43.290793 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eee3459-78be-4b93-9404-65865183177d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:43.291188 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eee3459-78be-4b93-9404-65865183177d admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:43.291554 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eee3459-78be-4b93-9404-65865183177d admin admin] PATH_INFO: `/v3/projects/c6ce95ccfb2c41fb8255925082d67b4a` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:43.293011 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eee3459-78be-4b93-9404-65865183177d admin admin] RBAC: Authorizing `identity:delete_project(project_id=c6ce95ccfb2c41fb8255925082d67b4a)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:43.295368 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eee3459-78be-4b93-9404-65865183177d admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:43.363446 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=89,keystone:INSERT=2,keystone:DELETE=9 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:43.391370 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-2eee3459-78be-4b93-9404-65865183177d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c6ce95ccfb2c41fb8255925082d67b4a', 'request_id': 'req-2eee3459-78be-4b93-9404-65865183177d'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:43.416461 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-2eee3459-78be-4b93-9404-65865183177d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c6ce95ccfb2c41fb8255925082d67b4a', 'request_id': 'req-2eee3459-78be-4b93-9404-65865183177d'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:43.427159 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-2eee3459-78be-4b93-9404-65865183177d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c6ce95ccfb2c41fb8255925082d67b4a', 'request_id': 'req-2eee3459-78be-4b93-9404-65865183177d'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:43.441395 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-2eee3459-78be-4b93-9404-65865183177d admin admin] The token cache is being invalidate because project c6ce95ccfb2c41fb8255925082d67b4a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 01:03:43.441777 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-2eee3459-78be-4b93-9404-65865183177d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:43.442747 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-2eee3459-78be-4b93-9404-65865183177d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:43.444619 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1370/2726] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 01:03:43 2022] DELETE /identity/v3/projects/c6ce95ccfb2c41fb8255925082d67b4a => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:03:43.449085 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614f4f7b-b6ab-408b-9a86-9077ccffd17e service neutron] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:43.461637 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614f4f7b-b6ab-408b-9a86-9077ccffd17e service neutron] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:43.548301 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0740f9b8-1836-41ae-ac89-127b0cc83f13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:43.548971 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0740f9b8-1836-41ae-ac89-127b0cc83f13 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:43.549447 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0740f9b8-1836-41ae-ac89-127b0cc83f13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:43.550040 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0740f9b8-1836-41ae-ac89-127b0cc83f13 admin admin] PATH_INFO: `/v3/users/9fef266826ea4f55a4dee661b086e363` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:43.551113 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0740f9b8-1836-41ae-ac89-127b0cc83f13 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9fef266826ea4f55a4dee661b086e363)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:43.553533 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0740f9b8-1836-41ae-ac89-127b0cc83f13 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:43.562172 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:43.635728 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-0740f9b8-1836-41ae-ac89-127b0cc83f13 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9fef266826ea4f55a4dee661b086e363'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:43.649091 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-0740f9b8-1836-41ae-ac89-127b0cc83f13 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9fef266826ea4f55a4dee661b086e363'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:43.650134 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-0740f9b8-1836-41ae-ac89-127b0cc83f13 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9fef266826ea4f55a4dee661b086e363'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:43.665993 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-0740f9b8-1836-41ae-ac89-127b0cc83f13 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9fef266826ea4f55a4dee661b086e363'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:43.667286 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-0740f9b8-1836-41ae-ac89-127b0cc83f13 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9fef266826ea4f55a4dee661b086e363'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:43.695929 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1357/2727] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 01:03:43 2022] DELETE /identity/v3/users/9fef266826ea4f55a4dee661b086e363 => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:03:44.578551 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eee3459-78be-4b93-9404-65865183177d admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:44.582142 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-2eee3459-78be-4b93-9404-65865183177d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:44.720853 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eee3459-78be-4b93-9404-65865183177d admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:44.822175 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d65e5a7-f40a-4dbf-bab3-9ea5c0eb1977 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:44.823246 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d65e5a7-f40a-4dbf-bab3-9ea5c0eb1977 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:44.823748 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d65e5a7-f40a-4dbf-bab3-9ea5c0eb1977 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:44.824227 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d65e5a7-f40a-4dbf-bab3-9ea5c0eb1977 admin admin] PATH_INFO: `/v3/projects/4b951ae6cc5843f2b6af8555d739bdf2` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:44.826452 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d65e5a7-f40a-4dbf-bab3-9ea5c0eb1977 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4b951ae6cc5843f2b6af8555d739bdf2)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:44.830148 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d65e5a7-f40a-4dbf-bab3-9ea5c0eb1977 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:44.927212 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-4d65e5a7-f40a-4dbf-bab3-9ea5c0eb1977 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4b951ae6cc5843f2b6af8555d739bdf2', 'request_id': 'req-4d65e5a7-f40a-4dbf-bab3-9ea5c0eb1977'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:44.936933 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-4d65e5a7-f40a-4dbf-bab3-9ea5c0eb1977 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4b951ae6cc5843f2b6af8555d739bdf2', 'request_id': 'req-4d65e5a7-f40a-4dbf-bab3-9ea5c0eb1977'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:44.943456 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=87,keystone:DELETE=11,keystone:INSERT=2 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:44.946400 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-4d65e5a7-f40a-4dbf-bab3-9ea5c0eb1977 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4b951ae6cc5843f2b6af8555d739bdf2', 'request_id': 'req-4d65e5a7-f40a-4dbf-bab3-9ea5c0eb1977'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:44.963992 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-4d65e5a7-f40a-4dbf-bab3-9ea5c0eb1977 admin admin] The token cache is being invalidate because project 4b951ae6cc5843f2b6af8555d739bdf2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 01:03:44.964913 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-4d65e5a7-f40a-4dbf-bab3-9ea5c0eb1977 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:44.966411 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-4d65e5a7-f40a-4dbf-bab3-9ea5c0eb1977 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:44.974199 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1371/2728] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 01:03:44 2022] DELETE /identity/v3/projects/4b951ae6cc5843f2b6af8555d739bdf2 => generated 0 bytes in 393 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:03:44.977465 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67f3bd7-a800-42b1-ad7d-2c308249ce73 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:44.978250 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67f3bd7-a800-42b1-ad7d-2c308249ce73 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:44.978396 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67f3bd7-a800-42b1-ad7d-2c308249ce73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:45.083859 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-c67f3bd7-a800-42b1-ad7d-2c308249ce73 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:03:45.085565 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-c67f3bd7-a800-42b1-ad7d-2c308249ce73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:45.213452 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1358/2729] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:03:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 235 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:03:45.224877 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af210d47-9461-4af6-b064-7be21c1003ee None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:45.226020 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af210d47-9461-4af6-b064-7be21c1003ee None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:45.226209 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af210d47-9461-4af6-b064-7be21c1003ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:45.262412 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-af210d47-9461-4af6-b064-7be21c1003ee None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:03:45.265984 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-af210d47-9461-4af6-b064-7be21c1003ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:45.287914 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1372/2730] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:03:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 61 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 01:03:45.289482 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67f3bd7-a800-42b1-ad7d-2c308249ce73 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:45.306724 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67f3bd7-a800-42b1-ad7d-2c308249ce73 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:45.322397 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-269796ce-7419-41df-ba06-dbdd603a6562 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:45.323483 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269796ce-7419-41df-ba06-dbdd603a6562 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:45.323922 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269796ce-7419-41df-ba06-dbdd603a6562 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:45.324384 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269796ce-7419-41df-ba06-dbdd603a6562 admin admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:45.326211 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-269796ce-7419-41df-ba06-dbdd603a6562 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 01:03:45.326427 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-269796ce-7419-41df-ba06-dbdd603a6562 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:45.329365 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-269796ce-7419-41df-ba06-dbdd603a6562 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:45.357885 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1359/2731] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 01:03:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 01:03:45.361922 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af210d47-9461-4af6-b064-7be21c1003ee None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:45.388110 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af210d47-9461-4af6-b064-7be21c1003ee None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:45.402924 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ad9756-af2e-46a9-8e68-d09e09216642 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:45.403490 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ad9756-af2e-46a9-8e68-d09e09216642 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:45.404083 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ad9756-af2e-46a9-8e68-d09e09216642 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:45.404609 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ad9756-af2e-46a9-8e68-d09e09216642 admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:45.405618 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68ad9756-af2e-46a9-8e68-d09e09216642 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:45.407963 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68ad9756-af2e-46a9-8e68-d09e09216642 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:45.449313 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1373/2732] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 01:03:45 2022] POST /identity/v3/projects => generated 381 bytes in 88 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 01:03:45.452508 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-269796ce-7419-41df-ba06-dbdd603a6562 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:45.465575 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-269796ce-7419-41df-ba06-dbdd603a6562 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:45.529358 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:45.553017 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e6b9bc-2e6e-48d8-b682-dfc1abe52275 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:45.553749 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e6b9bc-2e6e-48d8-b682-dfc1abe52275 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:45.554120 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e6b9bc-2e6e-48d8-b682-dfc1abe52275 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:45.554429 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e6b9bc-2e6e-48d8-b682-dfc1abe52275 admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:45.555182 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3e6b9bc-2e6e-48d8-b682-dfc1abe52275 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:45.557433 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3e6b9bc-2e6e-48d8-b682-dfc1abe52275 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:45.590427 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1360/2733] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 01:03:45 2022] POST /identity/v3/users => generated 291 bytes in 134 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 01:03:45.591761 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ad9756-af2e-46a9-8e68-d09e09216642 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:45.604947 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ad9756-af2e-46a9-8e68-d09e09216642 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:45.611207 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9bf3d6-a0c1-46ff-848d-3a5a465746de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:45.611578 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9bf3d6-a0c1-46ff-848d-3a5a465746de admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:45.612781 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9bf3d6-a0c1-46ff-848d-3a5a465746de admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:45.613287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9bf3d6-a0c1-46ff-848d-3a5a465746de admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:45.614395 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee9bf3d6-a0c1-46ff-848d-3a5a465746de admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:45.616162 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee9bf3d6-a0c1-46ff-848d-3a5a465746de admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:45.629320 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1374/2734] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:03:45 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 01:03:45.634553 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e6b9bc-2e6e-48d8-b682-dfc1abe52275 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:45.648313 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e6b9bc-2e6e-48d8-b682-dfc1abe52275 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:45.654978 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5694fc53-54b1-41b9-abc6-ae902ca63974 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:45.656075 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5694fc53-54b1-41b9-abc6-ae902ca63974 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:45.656610 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5694fc53-54b1-41b9-abc6-ae902ca63974 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:45.657157 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5694fc53-54b1-41b9-abc6-ae902ca63974 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:45.657960 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5694fc53-54b1-41b9-abc6-ae902ca63974 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:45.661307 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5694fc53-54b1-41b9-abc6-ae902ca63974 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:45.681144 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1361/2735] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:03:45 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 01:03:45.686660 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9bf3d6-a0c1-46ff-848d-3a5a465746de admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:45.724563 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9bf3d6-a0c1-46ff-848d-3a5a465746de admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:45.732826 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69d54ae-05c7-4b99-96b5-1b865908d0b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:45.734652 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69d54ae-05c7-4b99-96b5-1b865908d0b0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:45.735187 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69d54ae-05c7-4b99-96b5-1b865908d0b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:45.736653 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69d54ae-05c7-4b99-96b5-1b865908d0b0 admin admin] PATH_INFO: `/v3/projects/24539ff7316f4afebd0bb6f5c2749d98/users/423e63ded2114b54862a492abf4ce5fb/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:45.767121 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f69d54ae-05c7-4b99-96b5-1b865908d0b0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=24539ff7316f4afebd0bb6f5c2749d98, user_id=423e63ded2114b54862a492abf4ce5fb, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:45.771583 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f69d54ae-05c7-4b99-96b5-1b865908d0b0 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:45.806500 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1375/2736] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 01:03:45 2022] PUT /identity/v3/projects/24539ff7316f4afebd0bb6f5c2749d98/users/423e63ded2114b54862a492abf4ce5fb/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:03:45.899138 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d263896e-cd94-4a74-98b9-aac4c4bfcd9a None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:45.899502 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d263896e-cd94-4a74-98b9-aac4c4bfcd9a None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:45.899673 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d263896e-cd94-4a74-98b9-aac4c4bfcd9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:45.939135 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-d263896e-cd94-4a74-98b9-aac4c4bfcd9a None None] MFA Rules not processed for user `423e63ded2114b54862a492abf4ce5fb`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:03:45.940486 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-d263896e-cd94-4a74-98b9-aac4c4bfcd9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:46.042617 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69d54ae-05c7-4b99-96b5-1b865908d0b0 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:46.047938 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1362/2737] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:03:45 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:03:46.239987 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69d54ae-05c7-4b99-96b5-1b865908d0b0 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:46.260804 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe8c88f0-c237-4a97-978b-a3777f80f6f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:46.262385 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8c88f0-c237-4a97-978b-a3777f80f6f0 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:46.262727 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8c88f0-c237-4a97-978b-a3777f80f6f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:46.263079 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8c88f0-c237-4a97-978b-a3777f80f6f0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:46.263997 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe8c88f0-c237-4a97-978b-a3777f80f6f0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:46.273001 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe8c88f0-c237-4a97-978b-a3777f80f6f0 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:46.341636 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1376/2738] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 01:03:46 2022] POST /identity/v3/projects => generated 381 bytes in 299 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 01:03:46.347567 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d263896e-cd94-4a74-98b9-aac4c4bfcd9a None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:46.387657 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d263896e-cd94-4a74-98b9-aac4c4bfcd9a None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:46.638318 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-165a8b46-6a46-4a09-81ef-038863e0812b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:46.639202 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165a8b46-6a46-4a09-81ef-038863e0812b admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:46.639604 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165a8b46-6a46-4a09-81ef-038863e0812b admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:46.640012 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165a8b46-6a46-4a09-81ef-038863e0812b admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:46.640961 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-165a8b46-6a46-4a09-81ef-038863e0812b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:46.644887 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-165a8b46-6a46-4a09-81ef-038863e0812b admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:46.693840 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1363/2739] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 01:03:46 2022] POST /identity/v3/users => generated 297 bytes in 348 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 01:03:46.694798 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe8c88f0-c237-4a97-978b-a3777f80f6f0 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:46.736068 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe8c88f0-c237-4a97-978b-a3777f80f6f0 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:46.748781 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b11e7cf-9ba1-42a0-ba8c-042fb60e9adb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:46.750857 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b11e7cf-9ba1-42a0-ba8c-042fb60e9adb admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:46.751920 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b11e7cf-9ba1-42a0-ba8c-042fb60e9adb admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:46.754477 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b11e7cf-9ba1-42a0-ba8c-042fb60e9adb admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:46.757697 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b11e7cf-9ba1-42a0-ba8c-042fb60e9adb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:46.764396 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b11e7cf-9ba1-42a0-ba8c-042fb60e9adb admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:46.797215 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1377/2740] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:03:46 2022] GET /identity/v3/roles => generated 1420 bytes in 103 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 01:03:46.803798 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-165a8b46-6a46-4a09-81ef-038863e0812b admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:46.831707 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-165a8b46-6a46-4a09-81ef-038863e0812b admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:46.862968 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3837b510-67b3-43b0-8c7f-c5a736287115 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:46.863911 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3837b510-67b3-43b0-8c7f-c5a736287115 admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:46.864478 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3837b510-67b3-43b0-8c7f-c5a736287115 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:46.864849 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3837b510-67b3-43b0-8c7f-c5a736287115 admin admin] PATH_INFO: `/v3/projects/d6cca8ceef2e4221ace15fb7b996b6aa/users/23e01edf341647d5acbba3cb17a7b5f6/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:46.932939 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3837b510-67b3-43b0-8c7f-c5a736287115 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d6cca8ceef2e4221ace15fb7b996b6aa, user_id=23e01edf341647d5acbba3cb17a7b5f6, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:46.942896 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3837b510-67b3-43b0-8c7f-c5a736287115 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:46.986186 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1364/2741] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 01:03:46 2022] PUT /identity/v3/projects/d6cca8ceef2e4221ace15fb7b996b6aa/users/23e01edf341647d5acbba3cb17a7b5f6/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:03:46.987913 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b11e7cf-9ba1-42a0-ba8c-042fb60e9adb admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:46.992912 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a13733-137c-4a37-95b3-2c3a99412530 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:46.993513 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a13733-137c-4a37-95b3-2c3a99412530 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:46.994126 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a13733-137c-4a37-95b3-2c3a99412530 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:47.004645 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b11e7cf-9ba1-42a0-ba8c-042fb60e9adb admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:47.053634 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:47.064632 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-d0a13733-137c-4a37-95b3-2c3a99412530 None None] MFA Rules not processed for user `1e515c9c8dcf4a1aa7dd74fb7655adf4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:03:47.068804 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-d0a13733-137c-4a37-95b3-2c3a99412530 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:47.106058 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e36d468-9352-4b5a-ba17-d6335745e02c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:47.107002 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e36d468-9352-4b5a-ba17-d6335745e02c admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:47.107463 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e36d468-9352-4b5a-ba17-d6335745e02c admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:47.107916 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e36d468-9352-4b5a-ba17-d6335745e02c admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:47.109771 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e36d468-9352-4b5a-ba17-d6335745e02c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:47.113958 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e36d468-9352-4b5a-ba17-d6335745e02c admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:47.121870 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:DELETE=1,keystone:SELECT=96,keystone:INSERT=3 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:47.149199 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1378/2742] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:03:46 2022] GET /identity/v3/roles => generated 1420 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 01:03:47.152141 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e36d468-9352-4b5a-ba17-d6335745e02c admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:47.176471 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1365/2743] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:03:46 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:03:47.188004 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e36d468-9352-4b5a-ba17-d6335745e02c admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:47.195650 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea908e0-3b45-4292-b75a-6aead297d685 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:47.196623 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0a13733-137c-4a37-95b3-2c3a99412530 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:47.196767 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea908e0-3b45-4292-b75a-6aead297d685 admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:47.197140 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea908e0-3b45-4292-b75a-6aead297d685 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:47.197629 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea908e0-3b45-4292-b75a-6aead297d685 admin admin] PATH_INFO: `/v3/projects/d6cca8ceef2e4221ace15fb7b996b6aa/users/23e01edf341647d5acbba3cb17a7b5f6/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:47.199269 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-d0a13733-137c-4a37-95b3-2c3a99412530 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:47.201378 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ea908e0-3b45-4292-b75a-6aead297d685 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d6cca8ceef2e4221ace15fb7b996b6aa, user_id=23e01edf341647d5acbba3cb17a7b5f6, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:47.204760 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ea908e0-3b45-4292-b75a-6aead297d685 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:47.217838 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1379/2744] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 01:03:47 2022] PUT /identity/v3/projects/d6cca8ceef2e4221ace15fb7b996b6aa/users/23e01edf341647d5acbba3cb17a7b5f6/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:03:47.231027 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce8ee8e-489e-4540-8198-bb0b5cf4593f None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:47.231389 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce8ee8e-489e-4540-8198-bb0b5cf4593f None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:47.231711 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce8ee8e-489e-4540-8198-bb0b5cf4593f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:47.275829 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-8ce8ee8e-489e-4540-8198-bb0b5cf4593f None None] MFA Rules not processed for user `23e01edf341647d5acbba3cb17a7b5f6`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:03:47.277672 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-8ce8ee8e-489e-4540-8198-bb0b5cf4593f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:47.436234 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0a13733-137c-4a37-95b3-2c3a99412530 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:47.454335 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1380/2745] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:03:47 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 224 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 01:03:47.470502 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce8ee8e-489e-4540-8198-bb0b5cf4593f None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:47.473907 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-8ce8ee8e-489e-4540-8198-bb0b5cf4593f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:47.513246 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=100 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:47.529887 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb19ea58-f3a9-4ee6-ac3f-f262efb5a9dd service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f142988a4dd14a5f96eacea45c046f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:47.530852 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb19ea58-f3a9-4ee6-ac3f-f262efb5a9dd service nova] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:47.532865 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb19ea58-f3a9-4ee6-ac3f-f262efb5a9dd service nova] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:47.533221 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb19ea58-f3a9-4ee6-ac3f-f262efb5a9dd service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:47.575989 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce8ee8e-489e-4540-8198-bb0b5cf4593f None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:47.582414 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2373001-4cf3-4d09-9f93-deee640c57a6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:47.583159 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2373001-4cf3-4d09-9f93-deee640c57a6 service cinder] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:47.583508 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2373001-4cf3-4d09-9f93-deee640c57a6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:47.583847 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2373001-4cf3-4d09-9f93-deee640c57a6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:47.600121 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2373001-4cf3-4d09-9f93-deee640c57a6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:47.601899 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2373001-4cf3-4d09-9f93-deee640c57a6 service cinder] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:47.615974 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1381/2746] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:03:47 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:03:47.620310 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb19ea58-f3a9-4ee6-ac3f-f262efb5a9dd service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:47.623326 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb19ea58-f3a9-4ee6-ac3f-f262efb5a9dd service nova] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:47.643852 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1366/2747] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:03:47 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 448 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:03:48.350668 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2373001-4cf3-4d09-9f93-deee640c57a6 service cinder] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:48.465187 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2373001-4cf3-4d09-9f93-deee640c57a6 service cinder] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:48.475078 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a61374-bd65-4428-893e-ab9bea562cd4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:48.476683 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a61374-bd65-4428-893e-ab9bea562cd4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:48.478719 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a61374-bd65-4428-893e-ab9bea562cd4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:48.479391 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a61374-bd65-4428-893e-ab9bea562cd4 admin admin] PATH_INFO: `/v3/users/423e63ded2114b54862a492abf4ce5fb` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:48.481392 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a61374-bd65-4428-893e-ab9bea562cd4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=423e63ded2114b54862a492abf4ce5fb)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:48.485675 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a61374-bd65-4428-893e-ab9bea562cd4 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:48.506190 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=98,keystone:INSERT=1,keystone:DELETE=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:48.599173 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-80a61374-bd65-4428-893e-ab9bea562cd4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '423e63ded2114b54862a492abf4ce5fb'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:48.600683 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-80a61374-bd65-4428-893e-ab9bea562cd4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '423e63ded2114b54862a492abf4ce5fb'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:48.615573 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-80a61374-bd65-4428-893e-ab9bea562cd4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '423e63ded2114b54862a492abf4ce5fb'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:48.616322 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-80a61374-bd65-4428-893e-ab9bea562cd4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '423e63ded2114b54862a492abf4ce5fb'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:48.626803 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-80a61374-bd65-4428-893e-ab9bea562cd4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '423e63ded2114b54862a492abf4ce5fb'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:48.661894 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1382/2748] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 01:03:48 2022] DELETE /identity/v3/users/423e63ded2114b54862a492abf4ce5fb => generated 0 bytes in 312 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:03:48.669740 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b76ca1-6cbb-4b02-8008-b39c9b98981c None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:48.673633 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b76ca1-6cbb-4b02-8008-b39c9b98981c None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:48.675402 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b76ca1-6cbb-4b02-8008-b39c9b98981c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:48.698618 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-b6b76ca1-6cbb-4b02-8008-b39c9b98981c None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:03:48.699926 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-b6b76ca1-6cbb-4b02-8008-b39c9b98981c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:48.795289 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1367/2749] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:03:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:03:48.808461 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a61374-bd65-4428-893e-ab9bea562cd4 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:48.811260 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-80a61374-bd65-4428-893e-ab9bea562cd4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:48.892034 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a61374-bd65-4428-893e-ab9bea562cd4 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:48.897340 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43eab67c-5663-487d-acf8-327bcd0e04e5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:48.898001 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43eab67c-5663-487d-acf8-327bcd0e04e5 service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:48.899031 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43eab67c-5663-487d-acf8-327bcd0e04e5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:48.899031 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43eab67c-5663-487d-acf8-327bcd0e04e5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:48.910701 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43eab67c-5663-487d-acf8-327bcd0e04e5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:48.912299 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43eab67c-5663-487d-acf8-327bcd0e04e5 service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:48.924397 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1383/2750] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:03:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:03:49.436957 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b76ca1-6cbb-4b02-8008-b39c9b98981c None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:49.438882 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-b6b76ca1-6cbb-4b02-8008-b39c9b98981c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:49.455228 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b76ca1-6cbb-4b02-8008-b39c9b98981c None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:49.463159 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ca53a8-df68-41da-bcdd-3704cdde3e73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:49.463825 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ca53a8-df68-41da-bcdd-3704cdde3e73 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:49.464099 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ca53a8-df68-41da-bcdd-3704cdde3e73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:49.464369 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ca53a8-df68-41da-bcdd-3704cdde3e73 admin admin] PATH_INFO: `/v3/projects/24539ff7316f4afebd0bb6f5c2749d98` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:49.466486 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1ca53a8-df68-41da-bcdd-3704cdde3e73 admin admin] RBAC: Authorizing `identity:delete_project(project_id=24539ff7316f4afebd0bb6f5c2749d98)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:49.468779 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1ca53a8-df68-41da-bcdd-3704cdde3e73 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:49.570262 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-a1ca53a8-df68-41da-bcdd-3704cdde3e73 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '24539ff7316f4afebd0bb6f5c2749d98', 'request_id': 'req-a1ca53a8-df68-41da-bcdd-3704cdde3e73'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:49.582099 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-a1ca53a8-df68-41da-bcdd-3704cdde3e73 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '24539ff7316f4afebd0bb6f5c2749d98', 'request_id': 'req-a1ca53a8-df68-41da-bcdd-3704cdde3e73'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:49.592278 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-a1ca53a8-df68-41da-bcdd-3704cdde3e73 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '24539ff7316f4afebd0bb6f5c2749d98', 'request_id': 'req-a1ca53a8-df68-41da-bcdd-3704cdde3e73'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:49.607937 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-a1ca53a8-df68-41da-bcdd-3704cdde3e73 admin admin] The token cache is being invalidate because project 24539ff7316f4afebd0bb6f5c2749d98 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81386) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 01:03:49.608225 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-a1ca53a8-df68-41da-bcdd-3704cdde3e73 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:49.609177 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-a1ca53a8-df68-41da-bcdd-3704cdde3e73 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:49.611344 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1368/2751] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 01:03:49 2022] DELETE /identity/v3/projects/24539ff7316f4afebd0bb6f5c2749d98 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:03:49.617200 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43eab67c-5663-487d-acf8-327bcd0e04e5 service neutron] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:49.640750 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43eab67c-5663-487d-acf8-327bcd0e04e5 service neutron] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:49.726887 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96302262-896d-43ce-a780-3f8d0819e99c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:49.727463 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96302262-896d-43ce-a780-3f8d0819e99c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:49.727735 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96302262-896d-43ce-a780-3f8d0819e99c admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:49.728003 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96302262-896d-43ce-a780-3f8d0819e99c admin admin] PATH_INFO: `/v3/users/23e01edf341647d5acbba3cb17a7b5f6` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:49.729497 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96302262-896d-43ce-a780-3f8d0819e99c admin admin] RBAC: Authorizing `identity:delete_user(user_id=23e01edf341647d5acbba3cb17a7b5f6)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:49.731468 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96302262-896d-43ce-a780-3f8d0819e99c admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:49.800144 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:49.821669 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-96302262-896d-43ce-a780-3f8d0819e99c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '23e01edf341647d5acbba3cb17a7b5f6'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:49.822664 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-96302262-896d-43ce-a780-3f8d0819e99c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '23e01edf341647d5acbba3cb17a7b5f6'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:49.838099 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-96302262-896d-43ce-a780-3f8d0819e99c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '23e01edf341647d5acbba3cb17a7b5f6'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:49.839180 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-96302262-896d-43ce-a780-3f8d0819e99c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '23e01edf341647d5acbba3cb17a7b5f6'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:49.854141 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-96302262-896d-43ce-a780-3f8d0819e99c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '23e01edf341647d5acbba3cb17a7b5f6'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:49.895182 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1384/2752] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 01:03:49 2022] DELETE /identity/v3/users/23e01edf341647d5acbba3cb17a7b5f6 => generated 0 bytes in 274 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:03:50.003506 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7887a414-1d39-4ebe-a11c-56f83d38df05 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:50.004404 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7887a414-1d39-4ebe-a11c-56f83d38df05 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:50.005242 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7887a414-1d39-4ebe-a11c-56f83d38df05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:50.045289 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-7887a414-1d39-4ebe-a11c-56f83d38df05 None None] MFA Rules not processed for user `1e515c9c8dcf4a1aa7dd74fb7655adf4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:03:50.047316 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-7887a414-1d39-4ebe-a11c-56f83d38df05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:50.070959 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:50.151117 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1369/2753] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:03:50 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 01:03:50.501041 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96302262-896d-43ce-a780-3f8d0819e99c admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:50.501041 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-96302262-896d-43ce-a780-3f8d0819e99c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:50.654331 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96302262-896d-43ce-a780-3f8d0819e99c admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:50.665613 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8321427-c815-4241-9848-c36d1bcfa97b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:50.667932 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8321427-c815-4241-9848-c36d1bcfa97b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:50.668492 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8321427-c815-4241-9848-c36d1bcfa97b admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:50.669090 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8321427-c815-4241-9848-c36d1bcfa97b admin admin] PATH_INFO: `/v3/projects/d6cca8ceef2e4221ace15fb7b996b6aa` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:50.671454 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8321427-c815-4241-9848-c36d1bcfa97b admin admin] RBAC: Authorizing `identity:delete_project(project_id=d6cca8ceef2e4221ace15fb7b996b6aa)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:50.675259 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8321427-c815-4241-9848-c36d1bcfa97b admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:50.776378 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-d8321427-c815-4241-9848-c36d1bcfa97b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd6cca8ceef2e4221ace15fb7b996b6aa', 'request_id': 'req-d8321427-c815-4241-9848-c36d1bcfa97b'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:50.788293 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-d8321427-c815-4241-9848-c36d1bcfa97b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd6cca8ceef2e4221ace15fb7b996b6aa', 'request_id': 'req-d8321427-c815-4241-9848-c36d1bcfa97b'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:50.797885 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7887a414-1d39-4ebe-a11c-56f83d38df05 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:50.800368 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-7887a414-1d39-4ebe-a11c-56f83d38df05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:50.804255 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-d8321427-c815-4241-9848-c36d1bcfa97b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd6cca8ceef2e4221ace15fb7b996b6aa', 'request_id': 'req-d8321427-c815-4241-9848-c36d1bcfa97b'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:50.821738 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-d8321427-c815-4241-9848-c36d1bcfa97b admin admin] The token cache is being invalidate because project d6cca8ceef2e4221ace15fb7b996b6aa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 01:03:50.821738 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-d8321427-c815-4241-9848-c36d1bcfa97b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:50.822331 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-d8321427-c815-4241-9848-c36d1bcfa97b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:50.823892 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1385/2754] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 01:03:50 2022] DELETE /identity/v3/projects/d6cca8ceef2e4221ace15fb7b996b6aa => generated 0 bytes in 327 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:03:50.830083 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-570ff512-bb88-41a6-a6ca-ebaf0747a6cb None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:50.830488 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-570ff512-bb88-41a6-a6ca-ebaf0747a6cb None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:50.832823 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-570ff512-bb88-41a6-a6ca-ebaf0747a6cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:50.862682 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-570ff512-bb88-41a6-a6ca-ebaf0747a6cb None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:03:50.864929 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-570ff512-bb88-41a6-a6ca-ebaf0747a6cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:50.922571 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7887a414-1d39-4ebe-a11c-56f83d38df05 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:50.924718 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:50.967199 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1386/2755] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:03:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:03:50.984633 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b2ccaa-cd5f-4b27-8bb4-26f470a7ef2c None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:50.985313 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b2ccaa-cd5f-4b27-8bb4-26f470a7ef2c None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:50.985869 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b2ccaa-cd5f-4b27-8bb4-26f470a7ef2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:51.016254 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53825f6f-e432-4925-8802-314f3e21172f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:51.017014 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53825f6f-e432-4925-8802-314f3e21172f service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:51.017014 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53825f6f-e432-4925-8802-314f3e21172f service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:51.017371 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53825f6f-e432-4925-8802-314f3e21172f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:51.019524 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-53825f6f-e432-4925-8802-314f3e21172f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:51.024166 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-62b2ccaa-cd5f-4b27-8bb4-26f470a7ef2c None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:03:51.026510 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-62b2ccaa-cd5f-4b27-8bb4-26f470a7ef2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:51.030924 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53825f6f-e432-4925-8802-314f3e21172f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:51.034458 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53825f6f-e432-4925-8802-314f3e21172f service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:51.047852 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1387/2756] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:03:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 60 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 01:03:51.048132 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1370/2757] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:03:50 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 250 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 01:03:51.050853 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62b2ccaa-cd5f-4b27-8bb4-26f470a7ef2c None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:51.071738 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62b2ccaa-cd5f-4b27-8bb4-26f470a7ef2c None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:51.085344 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02b4b427-ed18-4573-9ee5-4979b59771a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:51.086612 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b4b427-ed18-4573-9ee5-4979b59771a4 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:51.087375 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b4b427-ed18-4573-9ee5-4979b59771a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:51.091421 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b4b427-ed18-4573-9ee5-4979b59771a4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:51.091421 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02b4b427-ed18-4573-9ee5-4979b59771a4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 01:03:51.092697 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02b4b427-ed18-4573-9ee5-4979b59771a4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:51.096892 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02b4b427-ed18-4573-9ee5-4979b59771a4 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:51.122348 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1388/2758] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 01:03:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 74 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 01:03:51.127434 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53825f6f-e432-4925-8802-314f3e21172f service neutron] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:51.145364 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53825f6f-e432-4925-8802-314f3e21172f service neutron] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:51.152125 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b8c4696-f446-4874-b81d-b30822aefdfb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:51.153215 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b8c4696-f446-4874-b81d-b30822aefdfb admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:51.153966 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b8c4696-f446-4874-b81d-b30822aefdfb admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:51.154692 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b8c4696-f446-4874-b81d-b30822aefdfb admin admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:51.155591 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b8c4696-f446-4874-b81d-b30822aefdfb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:51.159015 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b8c4696-f446-4874-b81d-b30822aefdfb admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:51.227237 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02b4b427-ed18-4573-9ee5-4979b59771a4 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:51.227678 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1371/2759] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 01:03:51 2022] POST /identity/v3/projects => generated 373 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 01:03:51.240912 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02b4b427-ed18-4573-9ee5-4979b59771a4 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:51.378962 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa30e12c-bb6b-4e68-be23-2051cccab7e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:51.379736 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa30e12c-bb6b-4e68-be23-2051cccab7e6 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:51.380144 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa30e12c-bb6b-4e68-be23-2051cccab7e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:51.380519 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa30e12c-bb6b-4e68-be23-2051cccab7e6 admin admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:51.381375 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa30e12c-bb6b-4e68-be23-2051cccab7e6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:51.383672 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa30e12c-bb6b-4e68-be23-2051cccab7e6 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:51.423013 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1389/2760] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 01:03:51 2022] POST /identity/v3/users => generated 287 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 01:03:51.427766 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b8c4696-f446-4874-b81d-b30822aefdfb admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:51.441845 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b8c4696-f446-4874-b81d-b30822aefdfb admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:51.449736 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d718af7c-a060-48f6-827f-e000b50be759 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:51.450604 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d718af7c-a060-48f6-827f-e000b50be759 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:51.450887 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d718af7c-a060-48f6-827f-e000b50be759 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:51.451167 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d718af7c-a060-48f6-827f-e000b50be759 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:51.451865 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d718af7c-a060-48f6-827f-e000b50be759 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:51.453904 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d718af7c-a060-48f6-827f-e000b50be759 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:51.482555 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa30e12c-bb6b-4e68-be23-2051cccab7e6 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:51.486113 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1372/2761] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:03:51 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 01:03:51.494807 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa30e12c-bb6b-4e68-be23-2051cccab7e6 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:51.525743 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328edc63-400d-4516-b085-8d249c4d3662 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:51.529364 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328edc63-400d-4516-b085-8d249c4d3662 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:51.529860 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328edc63-400d-4516-b085-8d249c4d3662 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:51.530329 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328edc63-400d-4516-b085-8d249c4d3662 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:51.538509 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328edc63-400d-4516-b085-8d249c4d3662 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:51.554740 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328edc63-400d-4516-b085-8d249c4d3662 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:51.572128 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1390/2762] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:03:51 2022] GET /identity/v3/roles => generated 1420 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 01:03:51.577300 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d718af7c-a060-48f6-827f-e000b50be759 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:51.590479 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d718af7c-a060-48f6-827f-e000b50be759 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:51.596742 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907bbc62-1533-4494-b19b-a9e039a44336 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:51.597595 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907bbc62-1533-4494-b19b-a9e039a44336 admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:51.598086 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907bbc62-1533-4494-b19b-a9e039a44336 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:51.598720 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907bbc62-1533-4494-b19b-a9e039a44336 admin admin] PATH_INFO: `/v3/projects/a4c77d3ed756497782dfc2b6a39ead1d/users/2e9b98acac74461e9c1cfe2625f2a7a5/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:51.637116 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907bbc62-1533-4494-b19b-a9e039a44336 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a4c77d3ed756497782dfc2b6a39ead1d, user_id=2e9b98acac74461e9c1cfe2625f2a7a5, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:51.655161 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907bbc62-1533-4494-b19b-a9e039a44336 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:51.710864 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1373/2763] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 01:03:51 2022] PUT /identity/v3/projects/a4c77d3ed756497782dfc2b6a39ead1d/users/2e9b98acac74461e9c1cfe2625f2a7a5/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:03:51.722139 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbb2f08-e8d4-49d1-8829-4e4496c3ad09 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:51.722776 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbb2f08-e8d4-49d1-8829-4e4496c3ad09 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:51.723113 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbb2f08-e8d4-49d1-8829-4e4496c3ad09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:51.783186 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-ccbb2f08-e8d4-49d1-8829-4e4496c3ad09 None None] MFA Rules not processed for user `2e9b98acac74461e9c1cfe2625f2a7a5`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:03:51.787510 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-ccbb2f08-e8d4-49d1-8829-4e4496c3ad09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:51.939564 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1391/2764] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:03:51 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:03:51.941081 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907bbc62-1533-4494-b19b-a9e039a44336 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:51.980618 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:52.047795 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907bbc62-1533-4494-b19b-a9e039a44336 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:52.074054 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc1f9010-cd23-4f67-b4b4-6e54ff82682b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:52.074626 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc1f9010-cd23-4f67-b4b4-6e54ff82682b admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:52.074951 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc1f9010-cd23-4f67-b4b4-6e54ff82682b admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:52.076674 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc1f9010-cd23-4f67-b4b4-6e54ff82682b admin admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:52.076674 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc1f9010-cd23-4f67-b4b4-6e54ff82682b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:52.080413 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc1f9010-cd23-4f67-b4b4-6e54ff82682b admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:52.140800 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1374/2765] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 01:03:51 2022] POST /identity/v3/projects => generated 373 bytes in 200 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 01:03:52.145712 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccbb2f08-e8d4-49d1-8829-4e4496c3ad09 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:52.184413 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccbb2f08-e8d4-49d1-8829-4e4496c3ad09 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:52.294214 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:52.359663 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba3e4a80-485a-4883-8426-6493a52872b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:52.360590 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba3e4a80-485a-4883-8426-6493a52872b3 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:52.360967 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba3e4a80-485a-4883-8426-6493a52872b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:52.361378 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba3e4a80-485a-4883-8426-6493a52872b3 admin admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:52.364118 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba3e4a80-485a-4883-8426-6493a52872b3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:52.366652 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba3e4a80-485a-4883-8426-6493a52872b3 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:52.425302 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1392/2766] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 01:03:52 2022] POST /identity/v3/users => generated 293 bytes in 280 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 01:03:52.427359 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc1f9010-cd23-4f67-b4b4-6e54ff82682b admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:52.445478 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc1f9010-cd23-4f67-b4b4-6e54ff82682b admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:52.458689 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc73ce32-291c-4912-a435-248edf5b0eed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:52.459317 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc73ce32-291c-4912-a435-248edf5b0eed admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:52.459591 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc73ce32-291c-4912-a435-248edf5b0eed admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:52.459912 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc73ce32-291c-4912-a435-248edf5b0eed admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:52.460499 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc73ce32-291c-4912-a435-248edf5b0eed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:52.462642 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc73ce32-291c-4912-a435-248edf5b0eed admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:52.484466 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1375/2767] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:03:52 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 01:03:52.488172 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba3e4a80-485a-4883-8426-6493a52872b3 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:52.528060 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba3e4a80-485a-4883-8426-6493a52872b3 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:52.535369 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b5161c-5292-4d88-b1b4-84d2eb1dd899 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:52.538315 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b5161c-5292-4d88-b1b4-84d2eb1dd899 admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:52.538800 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b5161c-5292-4d88-b1b4-84d2eb1dd899 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:52.539810 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b5161c-5292-4d88-b1b4-84d2eb1dd899 admin admin] PATH_INFO: `/v3/projects/ce04fef56605480eb8c6aca09c3a82f2/users/9f8b69eb92794dc2a29f5cc13ed82620/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:52.571712 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b5161c-5292-4d88-b1b4-84d2eb1dd899 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ce04fef56605480eb8c6aca09c3a82f2, user_id=9f8b69eb92794dc2a29f5cc13ed82620, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:52.574798 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b5161c-5292-4d88-b1b4-84d2eb1dd899 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:52.597435 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1393/2768] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 01:03:52 2022] PUT /identity/v3/projects/ce04fef56605480eb8c6aca09c3a82f2/users/9f8b69eb92794dc2a29f5cc13ed82620/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:03:52.604503 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc73ce32-291c-4912-a435-248edf5b0eed admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:52.622537 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc73ce32-291c-4912-a435-248edf5b0eed admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:52.704042 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214a141f-0f4e-43a2-bec5-b6fd526e545d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:52.704691 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214a141f-0f4e-43a2-bec5-b6fd526e545d admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:52.704986 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214a141f-0f4e-43a2-bec5-b6fd526e545d admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:52.706010 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214a141f-0f4e-43a2-bec5-b6fd526e545d admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:52.706674 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-214a141f-0f4e-43a2-bec5-b6fd526e545d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:52.708299 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-214a141f-0f4e-43a2-bec5-b6fd526e545d admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:52.726063 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1376/2769] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:03:52 2022] GET /identity/v3/roles => generated 1420 bytes in 123 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 01:03:52.732240 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b5161c-5292-4d88-b1b4-84d2eb1dd899 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:52.746184 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b5161c-5292-4d88-b1b4-84d2eb1dd899 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:52.751184 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8bade91-2a7c-4070-a337-4eef003b7dfd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:52.751847 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8bade91-2a7c-4070-a337-4eef003b7dfd admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:52.752123 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8bade91-2a7c-4070-a337-4eef003b7dfd admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:52.752471 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8bade91-2a7c-4070-a337-4eef003b7dfd admin admin] PATH_INFO: `/v3/projects/ce04fef56605480eb8c6aca09c3a82f2/users/9f8b69eb92794dc2a29f5cc13ed82620/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:52.755504 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8bade91-2a7c-4070-a337-4eef003b7dfd admin admin] RBAC: Authorizing `identity:create_grant(project_id=ce04fef56605480eb8c6aca09c3a82f2, user_id=9f8b69eb92794dc2a29f5cc13ed82620, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:52.758841 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8bade91-2a7c-4070-a337-4eef003b7dfd admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:52.772991 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1394/2770] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 01:03:52 2022] PUT /identity/v3/projects/ce04fef56605480eb8c6aca09c3a82f2/users/9f8b69eb92794dc2a29f5cc13ed82620/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 42 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:03:52.785246 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982ac82a-921c-40ef-b3d1-9eca15cd6302 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:52.785577 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982ac82a-921c-40ef-b3d1-9eca15cd6302 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:52.785832 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982ac82a-921c-40ef-b3d1-9eca15cd6302 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:52.819482 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-982ac82a-921c-40ef-b3d1-9eca15cd6302 None None] MFA Rules not processed for user `9f8b69eb92794dc2a29f5cc13ed82620`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:03:52.821227 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-982ac82a-921c-40ef-b3d1-9eca15cd6302 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:52.919337 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1377/2771] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:03:52 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 01:03:52.933714 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8bade91-2a7c-4070-a337-4eef003b7dfd admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:52.935414 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-b8bade91-2a7c-4070-a337-4eef003b7dfd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:53.019587 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8bade91-2a7c-4070-a337-4eef003b7dfd admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:53.023747 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27476bf-1c21-4bf1-abaa-edc0820729b4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:53.024387 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27476bf-1c21-4bf1-abaa-edc0820729b4 service cinder] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:53.024668 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27476bf-1c21-4bf1-abaa-edc0820729b4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:53.024936 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27476bf-1c21-4bf1-abaa-edc0820729b4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:53.106745 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27476bf-1c21-4bf1-abaa-edc0820729b4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:53.108754 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27476bf-1c21-4bf1-abaa-edc0820729b4 service cinder] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:53.120284 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1395/2772] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:03:52 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 187 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:03:54.774852 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-982ac82a-921c-40ef-b3d1-9eca15cd6302 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:54.776362 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-982ac82a-921c-40ef-b3d1-9eca15cd6302 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:54.789815 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-982ac82a-921c-40ef-b3d1-9eca15cd6302 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:54.794010 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43304576-f41b-4057-a520-e4490c0f239f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:54.794509 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43304576-f41b-4057-a520-e4490c0f239f service cinder] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:54.794784 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43304576-f41b-4057-a520-e4490c0f239f service cinder] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:54.795056 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43304576-f41b-4057-a520-e4490c0f239f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:54.797477 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-43304576-f41b-4057-a520-e4490c0f239f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:54.832184 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:54.897287 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43304576-f41b-4057-a520-e4490c0f239f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:54.898822 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43304576-f41b-4057-a520-e4490c0f239f service cinder] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:54.912447 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1378/2773] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:03:54 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:03:55.907485 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27476bf-1c21-4bf1-abaa-edc0820729b4 service cinder] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:55.957523 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:55.985022 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27476bf-1c21-4bf1-abaa-edc0820729b4 service cinder] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:55.990162 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab198c3-33b1-4238-8d17-57ea21a837a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:55.990819 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab198c3-33b1-4238-8d17-57ea21a837a7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:55.991094 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab198c3-33b1-4238-8d17-57ea21a837a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:55.991361 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab198c3-33b1-4238-8d17-57ea21a837a7 admin admin] PATH_INFO: `/v3/users/2e9b98acac74461e9c1cfe2625f2a7a5` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:55.992567 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cab198c3-33b1-4238-8d17-57ea21a837a7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2e9b98acac74461e9c1cfe2625f2a7a5)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:55.994700 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cab198c3-33b1-4238-8d17-57ea21a837a7 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:56.053190 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-cab198c3-33b1-4238-8d17-57ea21a837a7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2e9b98acac74461e9c1cfe2625f2a7a5'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:56.054057 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-cab198c3-33b1-4238-8d17-57ea21a837a7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2e9b98acac74461e9c1cfe2625f2a7a5'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:56.063406 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-cab198c3-33b1-4238-8d17-57ea21a837a7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2e9b98acac74461e9c1cfe2625f2a7a5'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:56.064215 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-cab198c3-33b1-4238-8d17-57ea21a837a7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2e9b98acac74461e9c1cfe2625f2a7a5'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:56.073819 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-cab198c3-33b1-4238-8d17-57ea21a837a7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2e9b98acac74461e9c1cfe2625f2a7a5'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:56.094256 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1396/2774] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 01:03:55 2022] DELETE /identity/v3/users/2e9b98acac74461e9c1cfe2625f2a7a5 => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 01:03:56.100250 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055b47a5-deb8-40fd-87df-ba1f8b65662a None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:56.101078 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055b47a5-deb8-40fd-87df-ba1f8b65662a None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:56.101248 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055b47a5-deb8-40fd-87df-ba1f8b65662a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:56.125895 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-055b47a5-deb8-40fd-87df-ba1f8b65662a None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:03:56.128473 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-055b47a5-deb8-40fd-87df-ba1f8b65662a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:56.213494 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1379/2775] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:03:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:03:56.226205 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab198c3-33b1-4238-8d17-57ea21a837a7 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:56.227955 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-cab198c3-33b1-4238-8d17-57ea21a837a7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:56.315612 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab198c3-33b1-4238-8d17-57ea21a837a7 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:56.321323 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ec7a12-856a-47af-9bdf-041251a129b9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:56.322237 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ec7a12-856a-47af-9bdf-041251a129b9 service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:56.322653 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ec7a12-856a-47af-9bdf-041251a129b9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:56.323063 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ec7a12-856a-47af-9bdf-041251a129b9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:56.335327 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6ec7a12-856a-47af-9bdf-041251a129b9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:56.336866 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6ec7a12-856a-47af-9bdf-041251a129b9 service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:56.348535 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1397/2776] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:03:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:03:56.713939 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-055b47a5-deb8-40fd-87df-ba1f8b65662a None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:56.716709 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-055b47a5-deb8-40fd-87df-ba1f8b65662a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:56.738844 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-055b47a5-deb8-40fd-87df-ba1f8b65662a None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:56.747249 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a8a181f-4f58-409b-a05f-a7792f84f8d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:56.748283 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8a181f-4f58-409b-a05f-a7792f84f8d4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:56.748623 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8a181f-4f58-409b-a05f-a7792f84f8d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:56.749142 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8a181f-4f58-409b-a05f-a7792f84f8d4 admin admin] PATH_INFO: `/v3/projects/a4c77d3ed756497782dfc2b6a39ead1d` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:56.750895 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a8a181f-4f58-409b-a05f-a7792f84f8d4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a4c77d3ed756497782dfc2b6a39ead1d)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:56.754325 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a8a181f-4f58-409b-a05f-a7792f84f8d4 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:56.818750 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-2a8a181f-4f58-409b-a05f-a7792f84f8d4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a4c77d3ed756497782dfc2b6a39ead1d', 'request_id': 'req-2a8a181f-4f58-409b-a05f-a7792f84f8d4'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:56.830016 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-2a8a181f-4f58-409b-a05f-a7792f84f8d4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a4c77d3ed756497782dfc2b6a39ead1d', 'request_id': 'req-2a8a181f-4f58-409b-a05f-a7792f84f8d4'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:56.841207 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-2a8a181f-4f58-409b-a05f-a7792f84f8d4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a4c77d3ed756497782dfc2b6a39ead1d', 'request_id': 'req-2a8a181f-4f58-409b-a05f-a7792f84f8d4'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:56.863813 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-2a8a181f-4f58-409b-a05f-a7792f84f8d4 admin admin] The token cache is being invalidate because project a4c77d3ed756497782dfc2b6a39ead1d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81386) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 01:03:56.864230 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-2a8a181f-4f58-409b-a05f-a7792f84f8d4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:56.865158 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-2a8a181f-4f58-409b-a05f-a7792f84f8d4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:56.867489 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1380/2777] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 01:03:56 2022] DELETE /identity/v3/projects/a4c77d3ed756497782dfc2b6a39ead1d => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 01:03:56.872762 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ec7a12-856a-47af-9bdf-041251a129b9 service neutron] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:56.884901 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ec7a12-856a-47af-9bdf-041251a129b9 service neutron] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:56.955083 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47c06ce-1030-4ec4-a0ff-86b538f310e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:56.955981 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47c06ce-1030-4ec4-a0ff-86b538f310e7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:56.956267 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47c06ce-1030-4ec4-a0ff-86b538f310e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:56.956541 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47c06ce-1030-4ec4-a0ff-86b538f310e7 admin admin] PATH_INFO: `/v3/users/9f8b69eb92794dc2a29f5cc13ed82620` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:56.957891 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47c06ce-1030-4ec4-a0ff-86b538f310e7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9f8b69eb92794dc2a29f5cc13ed82620)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:56.959870 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47c06ce-1030-4ec4-a0ff-86b538f310e7 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:56.961785 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:57.025465 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-b47c06ce-1030-4ec4-a0ff-86b538f310e7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9f8b69eb92794dc2a29f5cc13ed82620'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:57.026235 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-b47c06ce-1030-4ec4-a0ff-86b538f310e7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9f8b69eb92794dc2a29f5cc13ed82620'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:57.036166 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-b47c06ce-1030-4ec4-a0ff-86b538f310e7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9f8b69eb92794dc2a29f5cc13ed82620'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:57.036911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-b47c06ce-1030-4ec4-a0ff-86b538f310e7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9f8b69eb92794dc2a29f5cc13ed82620'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:57.046263 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-b47c06ce-1030-4ec4-a0ff-86b538f310e7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9f8b69eb92794dc2a29f5cc13ed82620'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:57.067622 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1398/2778] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 01:03:56 2022] DELETE /identity/v3/users/9f8b69eb92794dc2a29f5cc13ed82620 => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 01:03:57.433069 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a8a181f-4f58-409b-a05f-a7792f84f8d4 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:57.435069 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-2a8a181f-4f58-409b-a05f-a7792f84f8d4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:57.462316 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a8a181f-4f58-409b-a05f-a7792f84f8d4 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:57.522230 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:57.550806 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88af7a4-f193-4310-bb67-fbef88812778 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:57.551465 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88af7a4-f193-4310-bb67-fbef88812778 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:57.551751 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88af7a4-f193-4310-bb67-fbef88812778 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:57.552025 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88af7a4-f193-4310-bb67-fbef88812778 admin admin] PATH_INFO: `/v3/projects/ce04fef56605480eb8c6aca09c3a82f2` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:57.553392 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88af7a4-f193-4310-bb67-fbef88812778 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ce04fef56605480eb8c6aca09c3a82f2)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:57.555577 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88af7a4-f193-4310-bb67-fbef88812778 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:57.612386 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-b88af7a4-f193-4310-bb67-fbef88812778 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ce04fef56605480eb8c6aca09c3a82f2', 'request_id': 'req-b88af7a4-f193-4310-bb67-fbef88812778'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:57.622227 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-b88af7a4-f193-4310-bb67-fbef88812778 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ce04fef56605480eb8c6aca09c3a82f2', 'request_id': 'req-b88af7a4-f193-4310-bb67-fbef88812778'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:57.636826 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-b88af7a4-f193-4310-bb67-fbef88812778 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ce04fef56605480eb8c6aca09c3a82f2', 'request_id': 'req-b88af7a4-f193-4310-bb67-fbef88812778'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:57.657942 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-b88af7a4-f193-4310-bb67-fbef88812778 admin admin] The token cache is being invalidate because project ce04fef56605480eb8c6aca09c3a82f2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81386) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 01:03:57.658949 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-b88af7a4-f193-4310-bb67-fbef88812778 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:57.660013 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-b88af7a4-f193-4310-bb67-fbef88812778 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:03:57.662454 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1381/2779] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 01:03:57 2022] DELETE /identity/v3/projects/ce04fef56605480eb8c6aca09c3a82f2 => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:03:57.669234 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d99dbac-74eb-40a7-a700-ee8221ce8d0b None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:57.670058 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d99dbac-74eb-40a7-a700-ee8221ce8d0b None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:57.670888 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d99dbac-74eb-40a7-a700-ee8221ce8d0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:57.697005 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-9d99dbac-74eb-40a7-a700-ee8221ce8d0b None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:03:57.699284 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-9d99dbac-74eb-40a7-a700-ee8221ce8d0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:57.808142 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1399/2780] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:03:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 01:03:57.818876 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac02073-b6df-411d-8272-b28dfa4e21da None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:57.819213 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac02073-b6df-411d-8272-b28dfa4e21da None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:57.819611 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac02073-b6df-411d-8272-b28dfa4e21da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:57.843875 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-eac02073-b6df-411d-8272-b28dfa4e21da None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:03:57.845306 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-eac02073-b6df-411d-8272-b28dfa4e21da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:57.855190 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1382/2781] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:03:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 01:03:57.860694 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d99dbac-74eb-40a7-a700-ee8221ce8d0b None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:57.872874 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d99dbac-74eb-40a7-a700-ee8221ce8d0b None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:57.885241 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfbf5f1a-fe8d-4e12-9ce2-d62d208d90fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:57.885911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfbf5f1a-fe8d-4e12-9ce2-d62d208d90fc admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:57.886185 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfbf5f1a-fe8d-4e12-9ce2-d62d208d90fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:57.886449 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfbf5f1a-fe8d-4e12-9ce2-d62d208d90fc admin admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:57.887167 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfbf5f1a-fe8d-4e12-9ce2-d62d208d90fc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 01:03:57.887558 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfbf5f1a-fe8d-4e12-9ce2-d62d208d90fc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:57.890368 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfbf5f1a-fe8d-4e12-9ce2-d62d208d90fc admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:57.904716 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1400/2782] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 01:03:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 01:03:57.909893 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac02073-b6df-411d-8272-b28dfa4e21da None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:57.920528 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac02073-b6df-411d-8272-b28dfa4e21da None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:57.927289 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1640b4e-0310-4577-9fef-d8c5a67623f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:57.927924 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1640b4e-0310-4577-9fef-d8c5a67623f6 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:57.928204 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1640b4e-0310-4577-9fef-d8c5a67623f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:57.928452 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1640b4e-0310-4577-9fef-d8c5a67623f6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:57.929123 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1640b4e-0310-4577-9fef-d8c5a67623f6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:57.931108 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1640b4e-0310-4577-9fef-d8c5a67623f6 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:57.972905 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1383/2783] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 01:03:57 2022] POST /identity/v3/projects => generated 363 bytes in 64 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 01:03:57.977672 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfbf5f1a-fe8d-4e12-9ce2-d62d208d90fc admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:57.990722 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfbf5f1a-fe8d-4e12-9ce2-d62d208d90fc admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:58.059209 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb38d5e-9341-44ba-9c20-799190a9173d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:58.059733 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb38d5e-9341-44ba-9c20-799190a9173d admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:58.060011 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb38d5e-9341-44ba-9c20-799190a9173d admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:58.060281 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb38d5e-9341-44ba-9c20-799190a9173d admin admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:58.060953 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb38d5e-9341-44ba-9c20-799190a9173d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:58.062989 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb38d5e-9341-44ba-9c20-799190a9173d admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:58.075477 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:03:58.102412 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1401/2784] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 01:03:57 2022] POST /identity/v3/users => generated 282 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 01:03:58.104035 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1640b4e-0310-4577-9fef-d8c5a67623f6 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:58.116527 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1640b4e-0310-4577-9fef-d8c5a67623f6 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:58.122631 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8213fa2f-b644-4f72-8229-d9e491337b82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:58.123573 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8213fa2f-b644-4f72-8229-d9e491337b82 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:58.123888 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8213fa2f-b644-4f72-8229-d9e491337b82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:58.124633 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8213fa2f-b644-4f72-8229-d9e491337b82 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:58.126015 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8213fa2f-b644-4f72-8229-d9e491337b82 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:58.130784 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8213fa2f-b644-4f72-8229-d9e491337b82 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:58.154150 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1384/2785] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:03:58 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 01:03:58.164997 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb38d5e-9341-44ba-9c20-799190a9173d admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:58.197448 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb38d5e-9341-44ba-9c20-799190a9173d admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:58.206257 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326a445a-14b4-4107-8882-84111d9ac5c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:58.207441 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326a445a-14b4-4107-8882-84111d9ac5c7 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:58.207952 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326a445a-14b4-4107-8882-84111d9ac5c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:58.208504 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326a445a-14b4-4107-8882-84111d9ac5c7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:58.210384 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-326a445a-14b4-4107-8882-84111d9ac5c7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:58.213964 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-326a445a-14b4-4107-8882-84111d9ac5c7 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:58.247276 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1402/2786] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:03:58 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 01:03:58.250456 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8213fa2f-b644-4f72-8229-d9e491337b82 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:58.274152 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8213fa2f-b644-4f72-8229-d9e491337b82 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:58.284333 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f537cf-c737-4f7a-8dbf-36f38b35793a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:58.284333 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f537cf-c737-4f7a-8dbf-36f38b35793a admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:58.284333 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f537cf-c737-4f7a-8dbf-36f38b35793a admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:58.284333 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f537cf-c737-4f7a-8dbf-36f38b35793a admin admin] PATH_INFO: `/v3/projects/7228453de1b4457b95497a5f27fe7c7a/users/1439deaa30874aa48ebc1a2cf7a5f440/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:58.307628 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f537cf-c737-4f7a-8dbf-36f38b35793a admin admin] RBAC: Authorizing `identity:create_grant(project_id=7228453de1b4457b95497a5f27fe7c7a, user_id=1439deaa30874aa48ebc1a2cf7a5f440, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:58.311735 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f537cf-c737-4f7a-8dbf-36f38b35793a admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:58.345369 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1385/2787] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 01:03:58 2022] PUT /identity/v3/projects/7228453de1b4457b95497a5f27fe7c7a/users/1439deaa30874aa48ebc1a2cf7a5f440/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 88 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:03:58.349144 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450c9f31-0ed4-428d-ae36-cc07d690cd70 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:58.349825 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450c9f31-0ed4-428d-ae36-cc07d690cd70 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:58.350265 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450c9f31-0ed4-428d-ae36-cc07d690cd70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:58.407306 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-450c9f31-0ed4-428d-ae36-cc07d690cd70 None None] MFA Rules not processed for user `1439deaa30874aa48ebc1a2cf7a5f440`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:03:58.410388 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-450c9f31-0ed4-428d-ae36-cc07d690cd70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:58.581512 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1403/2788] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:03:58 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 234 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:03:58.598970 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f537cf-c737-4f7a-8dbf-36f38b35793a admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:03:58.601788 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-b7f537cf-c737-4f7a-8dbf-36f38b35793a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:03:58.716462 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f537cf-c737-4f7a-8dbf-36f38b35793a admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:03:58.724073 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f76951a6-82a2-45ef-9add-d5d547c11c76 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:03:58.725470 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f76951a6-82a2-45ef-9add-d5d547c11c76 service cinder] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:03:58.726130 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f76951a6-82a2-45ef-9add-d5d547c11c76 service cinder] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:03:58.726660 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f76951a6-82a2-45ef-9add-d5d547c11c76 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:03:58.741577 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f76951a6-82a2-45ef-9add-d5d547c11c76 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:03:58.744452 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f76951a6-82a2-45ef-9add-d5d547c11c76 service cinder] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:03:58.758061 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1386/2789] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:03:58 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 160 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 01:04:02.334580 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8afabb-16eb-4815-9882-3cc2e2cd1cdf None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:04:02.335379 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8afabb-16eb-4815-9882-3cc2e2cd1cdf None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:04:02.335530 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8afabb-16eb-4815-9882-3cc2e2cd1cdf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:04:02.362486 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-3d8afabb-16eb-4815-9882-3cc2e2cd1cdf None None] MFA Rules not processed for user `c19b10ebc31244dbb2b1539d94ac281d`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:04:02.364555 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-3d8afabb-16eb-4815-9882-3cc2e2cd1cdf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:04:02.458133 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1404/2790] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:04:02 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:04:04.666968 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f76951a6-82a2-45ef-9add-d5d547c11c76 service cinder] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:04:04.669474 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-f76951a6-82a2-45ef-9add-d5d547c11c76 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:04:04.704163 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:04:04.802871 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f76951a6-82a2-45ef-9add-d5d547c11c76 service cinder] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:04:04.808205 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d08227-b031-491d-81ba-2eaccc673d9f service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '11a80b55dcd444df901c8bf80f2a2c26', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:04:04.808829 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d08227-b031-491d-81ba-2eaccc673d9f service placement] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:04:04.809098 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d08227-b031-491d-81ba-2eaccc673d9f service placement] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:04:04.809401 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d08227-b031-491d-81ba-2eaccc673d9f service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:04:04.811247 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-96d08227-b031-491d-81ba-2eaccc673d9f service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:04:04.825088 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d08227-b031-491d-81ba-2eaccc673d9f service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:04:04.826737 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d08227-b031-491d-81ba-2eaccc673d9f service placement] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:04:04.841957 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1387/2791] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:04:04 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 175 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:04:12.450334 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=64 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:04:14.833741 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=35 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:04:27.724233 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d8afabb-16eb-4815-9882-3cc2e2cd1cdf None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:04:27.725752 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-3d8afabb-16eb-4815-9882-3cc2e2cd1cdf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:04:27.861824 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d8afabb-16eb-4815-9882-3cc2e2cd1cdf None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:04:27.868006 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5c9b21-004c-4e01-bccf-a1af562a944e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:04:27.869308 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5c9b21-004c-4e01-bccf-a1af562a944e service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:04:27.869635 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5c9b21-004c-4e01-bccf-a1af562a944e service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:04:27.870155 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5c9b21-004c-4e01-bccf-a1af562a944e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:04:27.873321 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-3b5c9b21-004c-4e01-bccf-a1af562a944e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:04:27.951229 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b5c9b21-004c-4e01-bccf-a1af562a944e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:04:27.952843 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b5c9b21-004c-4e01-bccf-a1af562a944e service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:04:27.965791 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1405/2792] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:04:27 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 244 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 01:04:37.960546 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=59 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:04:39.916143 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d08227-b031-491d-81ba-2eaccc673d9f service placement] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:04:40.027593 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d08227-b031-491d-81ba-2eaccc673d9f service placement] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:04:40.037456 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80fbe7b-fe9c-430d-9b37-2768ca48119b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:04:40.038583 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80fbe7b-fe9c-430d-9b37-2768ca48119b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:04:40.038993 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80fbe7b-fe9c-430d-9b37-2768ca48119b admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:04:40.039379 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80fbe7b-fe9c-430d-9b37-2768ca48119b admin admin] PATH_INFO: `/v3/users/1439deaa30874aa48ebc1a2cf7a5f440` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:04:40.041191 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e80fbe7b-fe9c-430d-9b37-2768ca48119b admin admin] RBAC: Authorizing `identity:delete_user(user_id=1439deaa30874aa48ebc1a2cf7a5f440)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:04:40.043488 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e80fbe7b-fe9c-430d-9b37-2768ca48119b admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:04:40.130082 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-e80fbe7b-fe9c-430d-9b37-2768ca48119b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1439deaa30874aa48ebc1a2cf7a5f440'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:04:40.144643 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-e80fbe7b-fe9c-430d-9b37-2768ca48119b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1439deaa30874aa48ebc1a2cf7a5f440'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:04:40.145680 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-e80fbe7b-fe9c-430d-9b37-2768ca48119b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1439deaa30874aa48ebc1a2cf7a5f440'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:04:40.160325 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-e80fbe7b-fe9c-430d-9b37-2768ca48119b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1439deaa30874aa48ebc1a2cf7a5f440'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:04:40.161105 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-e80fbe7b-fe9c-430d-9b37-2768ca48119b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1439deaa30874aa48ebc1a2cf7a5f440'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:04:40.196401 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1388/2793] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 01:04:39 2022] DELETE /identity/v3/users/1439deaa30874aa48ebc1a2cf7a5f440 => generated 0 bytes in 281 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:04:40.201609 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42116ad1-695f-4d30-b20f-9b9092e49aa6 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:04:40.202543 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42116ad1-695f-4d30-b20f-9b9092e49aa6 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:04:40.202543 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42116ad1-695f-4d30-b20f-9b9092e49aa6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:04:40.231525 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-42116ad1-695f-4d30-b20f-9b9092e49aa6 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:04:40.233625 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-42116ad1-695f-4d30-b20f-9b9092e49aa6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:04:40.339747 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1406/2794] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:04:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 01:04:40.352260 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80fbe7b-fe9c-430d-9b37-2768ca48119b admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:04:40.353979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-e80fbe7b-fe9c-430d-9b37-2768ca48119b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:04:40.446810 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80fbe7b-fe9c-430d-9b37-2768ca48119b admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:04:40.451741 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84b5f45c-f869-48bf-af8b-3481d08e72a5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:04:40.452403 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b5f45c-f869-48bf-af8b-3481d08e72a5 service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:04:40.452678 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b5f45c-f869-48bf-af8b-3481d08e72a5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:04:40.452944 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b5f45c-f869-48bf-af8b-3481d08e72a5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:04:40.465133 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84b5f45c-f869-48bf-af8b-3481d08e72a5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:04:40.467086 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84b5f45c-f869-48bf-af8b-3481d08e72a5 service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:04:40.480828 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1389/2795] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:04:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:04:40.843391 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42116ad1-695f-4d30-b20f-9b9092e49aa6 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:04:40.846801 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-42116ad1-695f-4d30-b20f-9b9092e49aa6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:04:40.870440 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42116ad1-695f-4d30-b20f-9b9092e49aa6 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:04:40.879829 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4e6dbf-0814-4cc3-add9-4fb1670d2619 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:04:40.880734 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4e6dbf-0814-4cc3-add9-4fb1670d2619 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:04:40.881207 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4e6dbf-0814-4cc3-add9-4fb1670d2619 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:04:40.881608 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4e6dbf-0814-4cc3-add9-4fb1670d2619 admin admin] PATH_INFO: `/v3/projects/7228453de1b4457b95497a5f27fe7c7a` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:04:40.883123 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef4e6dbf-0814-4cc3-add9-4fb1670d2619 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7228453de1b4457b95497a5f27fe7c7a)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:04:40.885775 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef4e6dbf-0814-4cc3-add9-4fb1670d2619 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:04:40.959979 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-ef4e6dbf-0814-4cc3-add9-4fb1670d2619 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7228453de1b4457b95497a5f27fe7c7a', 'request_id': 'req-ef4e6dbf-0814-4cc3-add9-4fb1670d2619'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:04:40.969673 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-ef4e6dbf-0814-4cc3-add9-4fb1670d2619 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7228453de1b4457b95497a5f27fe7c7a', 'request_id': 'req-ef4e6dbf-0814-4cc3-add9-4fb1670d2619'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:04:40.977408 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-ef4e6dbf-0814-4cc3-add9-4fb1670d2619 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7228453de1b4457b95497a5f27fe7c7a', 'request_id': 'req-ef4e6dbf-0814-4cc3-add9-4fb1670d2619'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:04:40.992640 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-ef4e6dbf-0814-4cc3-add9-4fb1670d2619 admin admin] The token cache is being invalidate because project 7228453de1b4457b95497a5f27fe7c7a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 01:04:40.992873 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-ef4e6dbf-0814-4cc3-add9-4fb1670d2619 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:04:40.993781 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-ef4e6dbf-0814-4cc3-add9-4fb1670d2619 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:04:40.995558 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1407/2796] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 01:04:40 2022] DELETE /identity/v3/projects/7228453de1b4457b95497a5f27fe7c7a => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:04:50.474539 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=79,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:04:50.988647 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=54,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:05:52.298355 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84b5f45c-f869-48bf-af8b-3481d08e72a5 service neutron] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:05:52.300573 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-84b5f45c-f869-48bf-af8b-3481d08e72a5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:05:52.398605 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84b5f45c-f869-48bf-af8b-3481d08e72a5 service neutron] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:05:52.455393 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d5a4a5-ee03-4d69-a2a3-612632434773 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:05:52.456010 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d5a4a5-ee03-4d69-a2a3-612632434773 service cinder] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:05:52.456280 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d5a4a5-ee03-4d69-a2a3-612632434773 service cinder] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:05:52.456541 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d5a4a5-ee03-4d69-a2a3-612632434773 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:05:52.458371 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-36d5a4a5-ee03-4d69-a2a3-612632434773 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:05:52.541906 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36d5a4a5-ee03-4d69-a2a3-612632434773 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:05:52.543372 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36d5a4a5-ee03-4d69-a2a3-612632434773 service cinder] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:05:52.557537 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1390/2797] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:05:52 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 263 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:05:55.406706 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4e6dbf-0814-4cc3-add9-4fb1670d2619 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:05:55.408912 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-ef4e6dbf-0814-4cc3-add9-4fb1670d2619 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:05:55.504679 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4e6dbf-0814-4cc3-add9-4fb1670d2619 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:05:55.508490 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d351990b-2569-4733-9866-eb640b1f92c4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:05:55.509113 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d351990b-2569-4733-9866-eb640b1f92c4 service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:05:55.509500 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d351990b-2569-4733-9866-eb640b1f92c4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:05:55.509772 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d351990b-2569-4733-9866-eb640b1f92c4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:05:55.511624 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-d351990b-2569-4733-9866-eb640b1f92c4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:05:55.527844 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d351990b-2569-4733-9866-eb640b1f92c4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:05:55.529569 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d351990b-2569-4733-9866-eb640b1f92c4 service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:05:55.543591 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1408/2798] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:05:55 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 01:06:02.552690 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=83 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:06:05.537498 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=26 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:06:23.307302 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d5a4a5-ee03-4d69-a2a3-612632434773 service cinder] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:06:23.309701 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-36d5a4a5-ee03-4d69-a2a3-612632434773 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:06:23.325090 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d5a4a5-ee03-4d69-a2a3-612632434773 service cinder] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:06:23.329837 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941a0ff0-f994-4304-a04b-4232b0b17826 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:06:23.330465 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941a0ff0-f994-4304-a04b-4232b0b17826 service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:06:23.330741 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941a0ff0-f994-4304-a04b-4232b0b17826 service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:06:23.331008 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941a0ff0-f994-4304-a04b-4232b0b17826 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:06:23.332950 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-941a0ff0-f994-4304-a04b-4232b0b17826 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:06:23.409525 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-941a0ff0-f994-4304-a04b-4232b0b17826 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:06:23.412369 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-941a0ff0-f994-4304-a04b-4232b0b17826 service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:06:23.432354 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1391/2799] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:06:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:06:31.028200 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d351990b-2569-4733-9866-eb640b1f92c4 service neutron] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:06:31.029990 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-d351990b-2569-4733-9866-eb640b1f92c4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:06:31.045690 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d351990b-2569-4733-9866-eb640b1f92c4 service neutron] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:06:31.050758 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fec3e43-a261-4029-8866-58daa5e4a449 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:06:31.051530 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fec3e43-a261-4029-8866-58daa5e4a449 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:06:31.051695 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fec3e43-a261-4029-8866-58daa5e4a449 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:06:31.051942 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fec3e43-a261-4029-8866-58daa5e4a449 admin admin] PATH_INFO: `/v3/users/c19b10ebc31244dbb2b1539d94ac281d` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:06:31.053110 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fec3e43-a261-4029-8866-58daa5e4a449 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c19b10ebc31244dbb2b1539d94ac281d)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:06:31.055105 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fec3e43-a261-4029-8866-58daa5e4a449 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:06:31.116515 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-6fec3e43-a261-4029-8866-58daa5e4a449 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c19b10ebc31244dbb2b1539d94ac281d'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:06:31.117409 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-6fec3e43-a261-4029-8866-58daa5e4a449 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c19b10ebc31244dbb2b1539d94ac281d'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:06:31.126921 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-6fec3e43-a261-4029-8866-58daa5e4a449 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c19b10ebc31244dbb2b1539d94ac281d'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:06:31.127628 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-6fec3e43-a261-4029-8866-58daa5e4a449 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c19b10ebc31244dbb2b1539d94ac281d'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:06:31.136457 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-6fec3e43-a261-4029-8866-58daa5e4a449 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c19b10ebc31244dbb2b1539d94ac281d'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:06:31.157330 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1409/2800] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 01:06:31 2022] DELETE /identity/v3/users/c19b10ebc31244dbb2b1539d94ac281d => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:06:31.362417 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941a0ff0-f994-4304-a04b-4232b0b17826 service neutron] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:06:31.378318 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941a0ff0-f994-4304-a04b-4232b0b17826 service neutron] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:06:31.447661 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b42442d3-1824-45cf-8ea2-f7f68a3a4cc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:06:31.448291 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42442d3-1824-45cf-8ea2-f7f68a3a4cc0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:06:31.448563 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42442d3-1824-45cf-8ea2-f7f68a3a4cc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:06:31.448824 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42442d3-1824-45cf-8ea2-f7f68a3a4cc0 admin admin] PATH_INFO: `/v3/projects/46f19fd9ad1e4880961d1eee75883687` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:06:31.450014 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b42442d3-1824-45cf-8ea2-f7f68a3a4cc0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=46f19fd9ad1e4880961d1eee75883687)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:06:31.452937 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b42442d3-1824-45cf-8ea2-f7f68a3a4cc0 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:06:31.508437 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-b42442d3-1824-45cf-8ea2-f7f68a3a4cc0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '46f19fd9ad1e4880961d1eee75883687', 'request_id': 'req-b42442d3-1824-45cf-8ea2-f7f68a3a4cc0'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:06:31.518997 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-b42442d3-1824-45cf-8ea2-f7f68a3a4cc0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '46f19fd9ad1e4880961d1eee75883687', 'request_id': 'req-b42442d3-1824-45cf-8ea2-f7f68a3a4cc0'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:06:31.526560 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-b42442d3-1824-45cf-8ea2-f7f68a3a4cc0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '46f19fd9ad1e4880961d1eee75883687', 'request_id': 'req-b42442d3-1824-45cf-8ea2-f7f68a3a4cc0'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:06:31.540719 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-b42442d3-1824-45cf-8ea2-f7f68a3a4cc0 admin admin] The token cache is being invalidate because project 46f19fd9ad1e4880961d1eee75883687 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81386) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 01:06:31.540990 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-b42442d3-1824-45cf-8ea2-f7f68a3a4cc0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:06:31.541743 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-b42442d3-1824-45cf-8ea2-f7f68a3a4cc0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:06:31.543290 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1392/2801] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 01:06:31 2022] DELETE /identity/v3/projects/46f19fd9ad1e4880961d1eee75883687 => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 01:06:31.549422 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fec3e43-a261-4029-8866-58daa5e4a449 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:06:31.551242 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-6fec3e43-a261-4029-8866-58daa5e4a449 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:06:31.571939 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fec3e43-a261-4029-8866-58daa5e4a449 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:06:31.680752 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ed8cb0-e24d-4292-93ac-918d20eaec43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:06:31.681876 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ed8cb0-e24d-4292-93ac-918d20eaec43 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:06:31.682369 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ed8cb0-e24d-4292-93ac-918d20eaec43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:06:31.682863 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ed8cb0-e24d-4292-93ac-918d20eaec43 admin admin] PATH_INFO: `/v3/users/919d4aec9bee4ef99aef27e7546bbe55` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:06:31.684900 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ed8cb0-e24d-4292-93ac-918d20eaec43 admin admin] RBAC: Authorizing `identity:delete_user(user_id=919d4aec9bee4ef99aef27e7546bbe55)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:06:31.688858 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ed8cb0-e24d-4292-93ac-918d20eaec43 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:06:31.745655 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-f0ed8cb0-e24d-4292-93ac-918d20eaec43 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '919d4aec9bee4ef99aef27e7546bbe55'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:06:31.746343 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-f0ed8cb0-e24d-4292-93ac-918d20eaec43 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '919d4aec9bee4ef99aef27e7546bbe55'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:06:31.756082 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-f0ed8cb0-e24d-4292-93ac-918d20eaec43 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '919d4aec9bee4ef99aef27e7546bbe55'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:06:31.756858 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-f0ed8cb0-e24d-4292-93ac-918d20eaec43 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '919d4aec9bee4ef99aef27e7546bbe55'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:06:31.767211 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-f0ed8cb0-e24d-4292-93ac-918d20eaec43 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '919d4aec9bee4ef99aef27e7546bbe55'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:06:31.775773 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=79,keystone:DELETE=19,keystone:INSERT=2 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:06:31.792327 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1410/2802] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 01:06:31 2022] DELETE /identity/v3/users/919d4aec9bee4ef99aef27e7546bbe55 => generated 0 bytes in 244 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:06:32.035015 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b42442d3-1824-45cf-8ea2-f7f68a3a4cc0 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:06:32.037123 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-b42442d3-1824-45cf-8ea2-f7f68a3a4cc0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:06:32.054292 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b42442d3-1824-45cf-8ea2-f7f68a3a4cc0 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:06:32.080912 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:06:32.127867 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c65e5998-bb15-4aea-98c3-2861f98cb353 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:06:32.128518 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65e5998-bb15-4aea-98c3-2861f98cb353 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:06:32.128813 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65e5998-bb15-4aea-98c3-2861f98cb353 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:06:32.129065 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65e5998-bb15-4aea-98c3-2861f98cb353 admin admin] PATH_INFO: `/v3/projects/adf4f7abd15d45d1a06405bdb1e94eef` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:06:32.130381 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c65e5998-bb15-4aea-98c3-2861f98cb353 admin admin] RBAC: Authorizing `identity:delete_project(project_id=adf4f7abd15d45d1a06405bdb1e94eef)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:06:32.132391 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c65e5998-bb15-4aea-98c3-2861f98cb353 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:06:32.194392 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-c65e5998-bb15-4aea-98c3-2861f98cb353 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'adf4f7abd15d45d1a06405bdb1e94eef', 'request_id': 'req-c65e5998-bb15-4aea-98c3-2861f98cb353'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:06:32.204214 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-c65e5998-bb15-4aea-98c3-2861f98cb353 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'adf4f7abd15d45d1a06405bdb1e94eef', 'request_id': 'req-c65e5998-bb15-4aea-98c3-2861f98cb353'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:06:32.210996 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-c65e5998-bb15-4aea-98c3-2861f98cb353 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'adf4f7abd15d45d1a06405bdb1e94eef', 'request_id': 'req-c65e5998-bb15-4aea-98c3-2861f98cb353'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:06:32.228396 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-c65e5998-bb15-4aea-98c3-2861f98cb353 admin admin] The token cache is being invalidate because project adf4f7abd15d45d1a06405bdb1e94eef was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81386) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 01:06:32.228924 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-c65e5998-bb15-4aea-98c3-2861f98cb353 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:06:32.230105 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-c65e5998-bb15-4aea-98c3-2861f98cb353 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:06:32.232857 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1393/2803] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 01:06:32 2022] DELETE /identity/v3/projects/adf4f7abd15d45d1a06405bdb1e94eef => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 01:06:41.791334 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=3,keystone:DELETE=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:06:42.221743 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=40,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:06:45.885083 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ed8cb0-e24d-4292-93ac-918d20eaec43 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:06:45.887281 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-f0ed8cb0-e24d-4292-93ac-918d20eaec43 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:06:46.057388 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ed8cb0-e24d-4292-93ac-918d20eaec43 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:06:46.190657 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c13e7786-217f-48da-b646-a102139bd0fa service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f142988a4dd14a5f96eacea45c046f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:06:46.191295 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13e7786-217f-48da-b646-a102139bd0fa service nova] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:06:46.192135 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13e7786-217f-48da-b646-a102139bd0fa service nova] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:06:46.192135 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13e7786-217f-48da-b646-a102139bd0fa service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:06:46.194266 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-c13e7786-217f-48da-b646-a102139bd0fa service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:06:46.212169 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c13e7786-217f-48da-b646-a102139bd0fa service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:06:46.221781 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c13e7786-217f-48da-b646-a102139bd0fa service nova] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:06:46.242295 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1411/2804] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:06:45 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 358 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:06:50.111260 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b58363-6ebb-40d5-972f-28dd544c74f6 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:06:50.112204 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b58363-6ebb-40d5-972f-28dd544c74f6 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:06:50.112204 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b58363-6ebb-40d5-972f-28dd544c74f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:06:50.133942 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-f4b58363-6ebb-40d5-972f-28dd544c74f6 None None] MFA Rules not processed for user `1e515c9c8dcf4a1aa7dd74fb7655adf4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:06:50.135174 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-f4b58363-6ebb-40d5-972f-28dd544c74f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:06:50.220943 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1394/2805] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:06:50 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 01:06:51.285851 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c13e7786-217f-48da-b646-a102139bd0fa service nova] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:06:51.288192 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-c13e7786-217f-48da-b646-a102139bd0fa service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:06:51.401585 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c13e7786-217f-48da-b646-a102139bd0fa service nova] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:06:51.410947 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7f4ad46-0d7f-4469-bb8a-52eb725bb3df service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '11a80b55dcd444df901c8bf80f2a2c26', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:06:51.411921 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f4ad46-0d7f-4469-bb8a-52eb725bb3df service placement] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:06:51.412378 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f4ad46-0d7f-4469-bb8a-52eb725bb3df service placement] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:06:51.412829 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f4ad46-0d7f-4469-bb8a-52eb725bb3df service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:06:51.415954 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-a7f4ad46-0d7f-4469-bb8a-52eb725bb3df service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:06:51.438402 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7f4ad46-0d7f-4469-bb8a-52eb725bb3df service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:06:51.441436 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7f4ad46-0d7f-4469-bb8a-52eb725bb3df service placement] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:06:51.460677 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1412/2806] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:06:51 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 176 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:06:53.838412 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b58363-6ebb-40d5-972f-28dd544c74f6 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:06:53.844010 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-f4b58363-6ebb-40d5-972f-28dd544c74f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:06:53.959555 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b58363-6ebb-40d5-972f-28dd544c74f6 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:06:53.964841 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1013e92e-cc59-4a02-9a27-878c291f44d4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:06:53.965554 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1013e92e-cc59-4a02-9a27-878c291f44d4 service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:06:53.965955 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1013e92e-cc59-4a02-9a27-878c291f44d4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:06:53.967214 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1013e92e-cc59-4a02-9a27-878c291f44d4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:06:53.969897 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-1013e92e-cc59-4a02-9a27-878c291f44d4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:06:53.981568 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1013e92e-cc59-4a02-9a27-878c291f44d4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:06:53.983132 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1013e92e-cc59-4a02-9a27-878c291f44d4 service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:06:53.993789 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1395/2807] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:06:53 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:06:56.935989 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7f4ad46-0d7f-4469-bb8a-52eb725bb3df service placement] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:06:56.937497 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-a7f4ad46-0d7f-4469-bb8a-52eb725bb3df service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:06:56.955504 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7f4ad46-0d7f-4469-bb8a-52eb725bb3df service placement] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:06:56.970637 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bee2cce-edf2-4ec3-8242-7938fb6df17e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f142988a4dd14a5f96eacea45c046f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:06:56.971726 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bee2cce-edf2-4ec3-8242-7938fb6df17e service nova] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:06:56.972418 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bee2cce-edf2-4ec3-8242-7938fb6df17e service nova] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:06:56.972819 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bee2cce-edf2-4ec3-8242-7938fb6df17e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:06:56.975128 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-8bee2cce-edf2-4ec3-8242-7938fb6df17e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:06:56.992846 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bee2cce-edf2-4ec3-8242-7938fb6df17e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:06:56.994679 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bee2cce-edf2-4ec3-8242-7938fb6df17e service nova] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:06:57.002239 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=100 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:06:57.024350 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1413/2808] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:06:56 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 01:06:59.540187 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1013e92e-cc59-4a02-9a27-878c291f44d4 service neutron] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:06:59.541780 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-1013e92e-cc59-4a02-9a27-878c291f44d4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:06:59.555185 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1013e92e-cc59-4a02-9a27-878c291f44d4 service neutron] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:06:59.559171 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ffde346-7b62-41d0-8fa4-7615048afe9a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:06:59.559791 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ffde346-7b62-41d0-8fa4-7615048afe9a service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:06:59.560099 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ffde346-7b62-41d0-8fa4-7615048afe9a service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:06:59.560397 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ffde346-7b62-41d0-8fa4-7615048afe9a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:06:59.562307 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-0ffde346-7b62-41d0-8fa4-7615048afe9a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:06:59.655641 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ffde346-7b62-41d0-8fa4-7615048afe9a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:06:59.658555 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ffde346-7b62-41d0-8fa4-7615048afe9a service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:06:59.679280 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1396/2809] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:06:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 01:07:02.069955 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bee2cce-edf2-4ec3-8242-7938fb6df17e service nova] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:07:02.072227 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-8bee2cce-edf2-4ec3-8242-7938fb6df17e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:07:02.089555 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bee2cce-edf2-4ec3-8242-7938fb6df17e service nova] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:07:02.098658 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeee74d6-c98a-4c98-a092-32f31c56dfd8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f142988a4dd14a5f96eacea45c046f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:07:02.099454 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeee74d6-c98a-4c98-a092-32f31c56dfd8 service nova] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:07:02.099874 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeee74d6-c98a-4c98-a092-32f31c56dfd8 service nova] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:07:02.100380 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeee74d6-c98a-4c98-a092-32f31c56dfd8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:07:02.104257 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-aeee74d6-c98a-4c98-a092-32f31c56dfd8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:07:02.121758 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeee74d6-c98a-4c98-a092-32f31c56dfd8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:07:02.124473 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeee74d6-c98a-4c98-a092-32f31c56dfd8 service nova] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:07:02.138340 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1414/2810] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:07:02 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 69 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:07:09.039727 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ffde346-7b62-41d0-8fa4-7615048afe9a service neutron] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:07:09.042084 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-0ffde346-7b62-41d0-8fa4-7615048afe9a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:07:09.063474 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ffde346-7b62-41d0-8fa4-7615048afe9a service neutron] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:07:09.071370 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4901080f-0739-44c7-872f-996872258ee7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:07:09.072584 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4901080f-0739-44c7-872f-996872258ee7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:07:09.073725 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4901080f-0739-44c7-872f-996872258ee7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:07:09.075171 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4901080f-0739-44c7-872f-996872258ee7 admin admin] PATH_INFO: `/v3/users/598ec83f7b7f4a9f90543218852aaf14` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:07:09.077950 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4901080f-0739-44c7-872f-996872258ee7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=598ec83f7b7f4a9f90543218852aaf14)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:07:09.080858 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4901080f-0739-44c7-872f-996872258ee7 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:07:09.153451 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:07:09.171386 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-4901080f-0739-44c7-872f-996872258ee7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '598ec83f7b7f4a9f90543218852aaf14'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:07:09.184845 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-4901080f-0739-44c7-872f-996872258ee7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '598ec83f7b7f4a9f90543218852aaf14'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:07:09.186301 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-4901080f-0739-44c7-872f-996872258ee7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '598ec83f7b7f4a9f90543218852aaf14'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:07:09.197502 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-4901080f-0739-44c7-872f-996872258ee7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '598ec83f7b7f4a9f90543218852aaf14'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:07:09.198472 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-4901080f-0739-44c7-872f-996872258ee7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '598ec83f7b7f4a9f90543218852aaf14'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:07:09.230008 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1397/2811] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 01:07:09 2022] DELETE /identity/v3/users/598ec83f7b7f4a9f90543218852aaf14 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:07:09.510177 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeee74d6-c98a-4c98-a092-32f31c56dfd8 service nova] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:07:09.524641 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeee74d6-c98a-4c98-a092-32f31c56dfd8 service nova] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:07:09.630937 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84634229-dec9-4f97-94e9-b32e790ecaa8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:07:09.631622 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84634229-dec9-4f97-94e9-b32e790ecaa8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:07:09.631919 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84634229-dec9-4f97-94e9-b32e790ecaa8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:07:09.632188 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84634229-dec9-4f97-94e9-b32e790ecaa8 admin admin] PATH_INFO: `/v3/projects/69a3d6ce251e480d866a86ed9f6f2eb9` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:07:09.633596 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84634229-dec9-4f97-94e9-b32e790ecaa8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=69a3d6ce251e480d866a86ed9f6f2eb9)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:07:09.635598 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84634229-dec9-4f97-94e9-b32e790ecaa8 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:07:09.698873 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-84634229-dec9-4f97-94e9-b32e790ecaa8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '69a3d6ce251e480d866a86ed9f6f2eb9', 'request_id': 'req-84634229-dec9-4f97-94e9-b32e790ecaa8'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:07:09.709442 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-84634229-dec9-4f97-94e9-b32e790ecaa8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '69a3d6ce251e480d866a86ed9f6f2eb9', 'request_id': 'req-84634229-dec9-4f97-94e9-b32e790ecaa8'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:07:09.718262 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-84634229-dec9-4f97-94e9-b32e790ecaa8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '69a3d6ce251e480d866a86ed9f6f2eb9', 'request_id': 'req-84634229-dec9-4f97-94e9-b32e790ecaa8'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:07:09.732918 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-84634229-dec9-4f97-94e9-b32e790ecaa8 admin admin] The token cache is being invalidate because project 69a3d6ce251e480d866a86ed9f6f2eb9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 01:07:09.733222 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-84634229-dec9-4f97-94e9-b32e790ecaa8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:07:09.734066 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-84634229-dec9-4f97-94e9-b32e790ecaa8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:07:09.735658 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1415/2812] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 01:07:09 2022] DELETE /identity/v3/projects/69a3d6ce251e480d866a86ed9f6f2eb9 => generated 0 bytes in 226 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:07:09.741361 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4901080f-0739-44c7-872f-996872258ee7 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:07:09.744014 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-4901080f-0739-44c7-872f-996872258ee7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:07:09.770388 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4901080f-0739-44c7-872f-996872258ee7 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:07:09.864265 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c122a17c-dea9-4256-84b9-866dc5ebb69e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:07:09.864916 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c122a17c-dea9-4256-84b9-866dc5ebb69e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:07:09.865194 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c122a17c-dea9-4256-84b9-866dc5ebb69e admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:07:09.865503 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c122a17c-dea9-4256-84b9-866dc5ebb69e admin admin] PATH_INFO: `/v3/users/de85ef7ef549463a88db6ff041f83574` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:07:09.866777 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c122a17c-dea9-4256-84b9-866dc5ebb69e admin admin] RBAC: Authorizing `identity:delete_user(user_id=de85ef7ef549463a88db6ff041f83574)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:07:09.868771 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c122a17c-dea9-4256-84b9-866dc5ebb69e admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:07:09.941109 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-c122a17c-dea9-4256-84b9-866dc5ebb69e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'de85ef7ef549463a88db6ff041f83574'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:07:09.955790 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-c122a17c-dea9-4256-84b9-866dc5ebb69e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'de85ef7ef549463a88db6ff041f83574'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:07:09.956552 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-c122a17c-dea9-4256-84b9-866dc5ebb69e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'de85ef7ef549463a88db6ff041f83574'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:07:09.966491 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-c122a17c-dea9-4256-84b9-866dc5ebb69e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'de85ef7ef549463a88db6ff041f83574'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:07:09.968731 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-c122a17c-dea9-4256-84b9-866dc5ebb69e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'de85ef7ef549463a88db6ff041f83574'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:07:09.992383 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1398/2813] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 01:07:09 2022] DELETE /identity/v3/users/de85ef7ef549463a88db6ff041f83574 => generated 0 bytes in 252 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:07:10.212703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84634229-dec9-4f97-94e9-b32e790ecaa8 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:07:10.214575 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-84634229-dec9-4f97-94e9-b32e790ecaa8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:07:10.230548 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84634229-dec9-4f97-94e9-b32e790ecaa8 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:07:10.307013 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b7a834-4792-47f3-a197-c9c84941c6c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:07:10.307753 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b7a834-4792-47f3-a197-c9c84941c6c8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:07:10.308074 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b7a834-4792-47f3-a197-c9c84941c6c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:07:10.308394 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b7a834-4792-47f3-a197-c9c84941c6c8 admin admin] PATH_INFO: `/v3/projects/2fcc4e24aaf24b7f8d61af98d0fd2431` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:07:10.310703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b7a834-4792-47f3-a197-c9c84941c6c8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2fcc4e24aaf24b7f8d61af98d0fd2431)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:07:10.312775 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b7a834-4792-47f3-a197-c9c84941c6c8 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:07:10.337412 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:07:10.386372 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-f7b7a834-4792-47f3-a197-c9c84941c6c8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2fcc4e24aaf24b7f8d61af98d0fd2431', 'request_id': 'req-f7b7a834-4792-47f3-a197-c9c84941c6c8'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:07:10.398679 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-f7b7a834-4792-47f3-a197-c9c84941c6c8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2fcc4e24aaf24b7f8d61af98d0fd2431', 'request_id': 'req-f7b7a834-4792-47f3-a197-c9c84941c6c8'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:07:10.410889 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-f7b7a834-4792-47f3-a197-c9c84941c6c8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2fcc4e24aaf24b7f8d61af98d0fd2431', 'request_id': 'req-f7b7a834-4792-47f3-a197-c9c84941c6c8'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:07:10.432218 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-f7b7a834-4792-47f3-a197-c9c84941c6c8 admin admin] The token cache is being invalidate because project 2fcc4e24aaf24b7f8d61af98d0fd2431 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 01:07:10.432814 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-f7b7a834-4792-47f3-a197-c9c84941c6c8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:07:10.433962 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-f7b7a834-4792-47f3-a197-c9c84941c6c8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:07:10.437460 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1416/2814] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 01:07:10 2022] DELETE /identity/v3/projects/2fcc4e24aaf24b7f8d61af98d0fd2431 => generated 0 bytes in 226 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:07:19.987486 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=68,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:07:20.426519 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=16,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:08:19.779146 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c122a17c-dea9-4256-84b9-866dc5ebb69e admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:19.781161 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-c122a17c-dea9-4256-84b9-866dc5ebb69e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:08:19.802423 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c122a17c-dea9-4256-84b9-866dc5ebb69e admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:19.874308 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ef6550-192b-466b-bc75-fcd2d98e3c9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:19.875029 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ef6550-192b-466b-bc75-fcd2d98e3c9b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:19.875407 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ef6550-192b-466b-bc75-fcd2d98e3c9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:19.875793 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ef6550-192b-466b-bc75-fcd2d98e3c9b admin admin] PATH_INFO: `/v3/users/1e515c9c8dcf4a1aa7dd74fb7655adf4` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:19.877217 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68ef6550-192b-466b-bc75-fcd2d98e3c9b admin admin] RBAC: Authorizing `identity:delete_user(user_id=1e515c9c8dcf4a1aa7dd74fb7655adf4)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:19.879386 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68ef6550-192b-466b-bc75-fcd2d98e3c9b admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:19.959385 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-68ef6550-192b-466b-bc75-fcd2d98e3c9b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1e515c9c8dcf4a1aa7dd74fb7655adf4'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:19.971216 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-68ef6550-192b-466b-bc75-fcd2d98e3c9b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1e515c9c8dcf4a1aa7dd74fb7655adf4'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:19.971911 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-68ef6550-192b-466b-bc75-fcd2d98e3c9b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1e515c9c8dcf4a1aa7dd74fb7655adf4'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:19.981136 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-68ef6550-192b-466b-bc75-fcd2d98e3c9b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1e515c9c8dcf4a1aa7dd74fb7655adf4'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:19.982189 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-68ef6550-192b-466b-bc75-fcd2d98e3c9b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1e515c9c8dcf4a1aa7dd74fb7655adf4'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:20.005905 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1399/2815] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 01:08:19 2022] DELETE /identity/v3/users/1e515c9c8dcf4a1aa7dd74fb7655adf4 => generated 0 bytes in 228 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:08:20.220906 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b7a834-4792-47f3-a197-c9c84941c6c8 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:20.223612 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-f7b7a834-4792-47f3-a197-c9c84941c6c8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:08:20.243441 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b7a834-4792-47f3-a197-c9c84941c6c8 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:20.328384 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8757f62-e4a1-41de-a7a5-d6d31b0dbba9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:20.329087 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8757f62-e4a1-41de-a7a5-d6d31b0dbba9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:20.329499 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8757f62-e4a1-41de-a7a5-d6d31b0dbba9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:20.329955 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8757f62-e4a1-41de-a7a5-d6d31b0dbba9 admin admin] PATH_INFO: `/v3/projects/f37b53a21b19482b8dcac6c38505a50b` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:20.331392 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8757f62-e4a1-41de-a7a5-d6d31b0dbba9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f37b53a21b19482b8dcac6c38505a50b)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:20.333650 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8757f62-e4a1-41de-a7a5-d6d31b0dbba9 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:20.398246 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-a8757f62-e4a1-41de-a7a5-d6d31b0dbba9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f37b53a21b19482b8dcac6c38505a50b', 'request_id': 'req-a8757f62-e4a1-41de-a7a5-d6d31b0dbba9'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:20.410348 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-a8757f62-e4a1-41de-a7a5-d6d31b0dbba9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f37b53a21b19482b8dcac6c38505a50b', 'request_id': 'req-a8757f62-e4a1-41de-a7a5-d6d31b0dbba9'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:20.420447 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-a8757f62-e4a1-41de-a7a5-d6d31b0dbba9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f37b53a21b19482b8dcac6c38505a50b', 'request_id': 'req-a8757f62-e4a1-41de-a7a5-d6d31b0dbba9'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:20.434845 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-a8757f62-e4a1-41de-a7a5-d6d31b0dbba9 admin admin] The token cache is being invalidate because project f37b53a21b19482b8dcac6c38505a50b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 01:08:20.435377 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-a8757f62-e4a1-41de-a7a5-d6d31b0dbba9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:20.436175 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-a8757f62-e4a1-41de-a7a5-d6d31b0dbba9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:20.438393 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1417/2816] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 01:08:20 2022] DELETE /identity/v3/projects/f37b53a21b19482b8dcac6c38505a50b => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:08:20.443860 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd56c9c-43a2-4c7d-99b0-f6d19082b46b None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:20.444309 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd56c9c-43a2-4c7d-99b0-f6d19082b46b None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:20.444764 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd56c9c-43a2-4c7d-99b0-f6d19082b46b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:20.472719 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-cfd56c9c-43a2-4c7d-99b0-f6d19082b46b None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:08:20.474355 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-cfd56c9c-43a2-4c7d-99b0-f6d19082b46b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:08:20.560850 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1400/2817] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:08:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 01:08:20.570067 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b22b08e-259f-4d2c-8004-c3966bace2ae None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:20.570487 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b22b08e-259f-4d2c-8004-c3966bace2ae None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:20.570875 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b22b08e-259f-4d2c-8004-c3966bace2ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:20.594746 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-2b22b08e-259f-4d2c-8004-c3966bace2ae None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:08:20.596013 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-2b22b08e-259f-4d2c-8004-c3966bace2ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:08:20.604436 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1418/2818] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:08:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 01:08:20.609965 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd56c9c-43a2-4c7d-99b0-f6d19082b46b None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:20.621212 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd56c9c-43a2-4c7d-99b0-f6d19082b46b None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:20.626349 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c3066f-cac4-4329-9086-1238682a7e86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:20.626922 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c3066f-cac4-4329-9086-1238682a7e86 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:20.627935 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c3066f-cac4-4329-9086-1238682a7e86 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:20.628214 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c3066f-cac4-4329-9086-1238682a7e86 admin admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:20.628887 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c3066f-cac4-4329-9086-1238682a7e86 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 01:08:20.629188 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c3066f-cac4-4329-9086-1238682a7e86 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:20.630942 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c3066f-cac4-4329-9086-1238682a7e86 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:20.645284 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1401/2819] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 01:08:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 01:08:20.650490 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b22b08e-259f-4d2c-8004-c3966bace2ae None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:20.660595 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b22b08e-259f-4d2c-8004-c3966bace2ae None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:20.665597 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc432c37-8e93-4ef9-9497-ba3d728526fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:20.666220 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc432c37-8e93-4ef9-9497-ba3d728526fe admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:20.666498 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc432c37-8e93-4ef9-9497-ba3d728526fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:20.666769 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc432c37-8e93-4ef9-9497-ba3d728526fe admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:20.667435 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc432c37-8e93-4ef9-9497-ba3d728526fe admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:20.670145 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc432c37-8e93-4ef9-9497-ba3d728526fe admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:20.712824 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1419/2820] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 01:08:20 2022] POST /identity/v3/projects => generated 363 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 01:08:20.719468 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c3066f-cac4-4329-9086-1238682a7e86 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:20.737138 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c3066f-cac4-4329-9086-1238682a7e86 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:20.743411 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:08:20.809291 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f72f47c-5b51-4e69-9e2b-40eb4ce881b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:20.810274 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f72f47c-5b51-4e69-9e2b-40eb4ce881b0 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:20.810274 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f72f47c-5b51-4e69-9e2b-40eb4ce881b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:20.810541 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f72f47c-5b51-4e69-9e2b-40eb4ce881b0 admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:20.811038 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f72f47c-5b51-4e69-9e2b-40eb4ce881b0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:20.812990 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f72f47c-5b51-4e69-9e2b-40eb4ce881b0 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:20.837149 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1402/2821] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 01:08:20 2022] POST /identity/v3/users => generated 282 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 01:08:20.842441 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc432c37-8e93-4ef9-9497-ba3d728526fe admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:20.853610 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc432c37-8e93-4ef9-9497-ba3d728526fe admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:20.858870 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f225eb36-396e-4f9c-8b80-3a48c9514e62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:20.859494 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f225eb36-396e-4f9c-8b80-3a48c9514e62 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:20.859786 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f225eb36-396e-4f9c-8b80-3a48c9514e62 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:20.860044 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f225eb36-396e-4f9c-8b80-3a48c9514e62 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:20.860712 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f225eb36-396e-4f9c-8b80-3a48c9514e62 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:20.862410 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f225eb36-396e-4f9c-8b80-3a48c9514e62 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:20.875139 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1420/2822] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:08:20 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 01:08:20.880872 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f72f47c-5b51-4e69-9e2b-40eb4ce881b0 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:20.894514 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f72f47c-5b51-4e69-9e2b-40eb4ce881b0 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:20.902962 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9c44641-906b-49c8-add2-65f66e7765c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:20.903881 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c44641-906b-49c8-add2-65f66e7765c2 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:20.904355 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c44641-906b-49c8-add2-65f66e7765c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:20.904806 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c44641-906b-49c8-add2-65f66e7765c2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:20.906006 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9c44641-906b-49c8-add2-65f66e7765c2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:20.909197 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9c44641-906b-49c8-add2-65f66e7765c2 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:20.930251 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1403/2823] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:08:20 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 01:08:20.936077 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f225eb36-396e-4f9c-8b80-3a48c9514e62 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:20.954077 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f225eb36-396e-4f9c-8b80-3a48c9514e62 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:20.961927 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27ea168a-dea1-47ba-ac9c-934114096f2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:20.963031 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ea168a-dea1-47ba-ac9c-934114096f2a admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:20.963508 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ea168a-dea1-47ba-ac9c-934114096f2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:20.963987 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ea168a-dea1-47ba-ac9c-934114096f2a admin admin] PATH_INFO: `/v3/projects/02094d9dfdc245d38c7181f92f9f6b86/users/9b5b6260e1a346da9c0c64c6f64b05c4/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:20.992179 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27ea168a-dea1-47ba-ac9c-934114096f2a admin admin] RBAC: Authorizing `identity:create_grant(project_id=02094d9dfdc245d38c7181f92f9f6b86, user_id=9b5b6260e1a346da9c0c64c6f64b05c4, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:20.998072 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27ea168a-dea1-47ba-ac9c-934114096f2a admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:21.020873 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1421/2824] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 01:08:20 2022] PUT /identity/v3/projects/02094d9dfdc245d38c7181f92f9f6b86/users/9b5b6260e1a346da9c0c64c6f64b05c4/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 86 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:08:21.033093 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1abeace7-d8ea-4290-a4c1-e7a0a9d72112 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:21.033447 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1abeace7-d8ea-4290-a4c1-e7a0a9d72112 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:21.033732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1abeace7-d8ea-4290-a4c1-e7a0a9d72112 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:21.066506 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-1abeace7-d8ea-4290-a4c1-e7a0a9d72112 None None] MFA Rules not processed for user `9b5b6260e1a346da9c0c64c6f64b05c4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:08:21.067771 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-1abeace7-d8ea-4290-a4c1-e7a0a9d72112 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:08:21.141567 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1404/2825] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:08:21 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 01:08:21.146366 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27ea168a-dea1-47ba-ac9c-934114096f2a admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:21.170623 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:08:21.226203 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27ea168a-dea1-47ba-ac9c-934114096f2a admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:21.232224 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e4f5d28-6ba7-4dd8-9a60-61bd18ad84ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:21.232852 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4f5d28-6ba7-4dd8-9a60-61bd18ad84ba admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:21.233200 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4f5d28-6ba7-4dd8-9a60-61bd18ad84ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:21.233424 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4f5d28-6ba7-4dd8-9a60-61bd18ad84ba admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:21.234065 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e4f5d28-6ba7-4dd8-9a60-61bd18ad84ba admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:21.236005 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e4f5d28-6ba7-4dd8-9a60-61bd18ad84ba admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:21.273144 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1422/2826] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 01:08:21 2022] POST /identity/v3/projects => generated 365 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 01:08:21.274460 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1abeace7-d8ea-4290-a4c1-e7a0a9d72112 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:21.287907 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1abeace7-d8ea-4290-a4c1-e7a0a9d72112 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:21.381928 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d4a3b15-1956-471b-bd0c-73d0645681bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:21.382920 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d4a3b15-1956-471b-bd0c-73d0645681bb admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:21.383402 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d4a3b15-1956-471b-bd0c-73d0645681bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:21.383875 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d4a3b15-1956-471b-bd0c-73d0645681bb admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:21.384983 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d4a3b15-1956-471b-bd0c-73d0645681bb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:21.389297 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d4a3b15-1956-471b-bd0c-73d0645681bb admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:21.422570 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1405/2827] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 01:08:21 2022] POST /identity/v3/users => generated 289 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 01:08:21.427450 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e4f5d28-6ba7-4dd8-9a60-61bd18ad84ba admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:21.439092 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e4f5d28-6ba7-4dd8-9a60-61bd18ad84ba admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:21.444079 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1bd8903-5cae-4dc9-bafe-7c242cc4c7e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:21.444703 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1bd8903-5cae-4dc9-bafe-7c242cc4c7e4 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:21.444981 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1bd8903-5cae-4dc9-bafe-7c242cc4c7e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:21.445251 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1bd8903-5cae-4dc9-bafe-7c242cc4c7e4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:21.445940 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1bd8903-5cae-4dc9-bafe-7c242cc4c7e4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:21.447566 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1bd8903-5cae-4dc9-bafe-7c242cc4c7e4 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:21.460667 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1423/2828] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:08:21 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 01:08:21.465941 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d4a3b15-1956-471b-bd0c-73d0645681bb admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:21.473414 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:08:21.478185 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d4a3b15-1956-471b-bd0c-73d0645681bb admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:21.487297 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46025d69-7162-4ad9-a9cf-39ee224002a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:21.488796 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46025d69-7162-4ad9-a9cf-39ee224002a6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:21.489118 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46025d69-7162-4ad9-a9cf-39ee224002a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:21.489478 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46025d69-7162-4ad9-a9cf-39ee224002a6 admin admin] PATH_INFO: `/v3/projects/7e340a45aec4486ca8a50e2f2f7e231e/users/b8c789926abe40ceb7c17b0180c07472/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:21.511942 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46025d69-7162-4ad9-a9cf-39ee224002a6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7e340a45aec4486ca8a50e2f2f7e231e, user_id=b8c789926abe40ceb7c17b0180c07472, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:21.514916 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46025d69-7162-4ad9-a9cf-39ee224002a6 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:21.534409 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1406/2829] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 01:08:21 2022] PUT /identity/v3/projects/7e340a45aec4486ca8a50e2f2f7e231e/users/b8c789926abe40ceb7c17b0180c07472/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 01:08:21.540960 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1bd8903-5cae-4dc9-bafe-7c242cc4c7e4 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:21.552934 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1bd8903-5cae-4dc9-bafe-7c242cc4c7e4 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:21.630158 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-712e65aa-90d9-40b2-9749-2f05a70867e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:21.631252 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712e65aa-90d9-40b2-9749-2f05a70867e6 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:21.631736 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712e65aa-90d9-40b2-9749-2f05a70867e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:21.632208 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712e65aa-90d9-40b2-9749-2f05a70867e6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:21.633364 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-712e65aa-90d9-40b2-9749-2f05a70867e6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:21.636375 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-712e65aa-90d9-40b2-9749-2f05a70867e6 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:21.656409 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1424/2830] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:08:21 2022] GET /identity/v3/roles => generated 1420 bytes in 116 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 01:08:21.660980 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46025d69-7162-4ad9-a9cf-39ee224002a6 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:21.672360 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46025d69-7162-4ad9-a9cf-39ee224002a6 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:21.677497 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd668c7-2729-4f3f-acef-c20a7baf76ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:21.678096 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd668c7-2729-4f3f-acef-c20a7baf76ed admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:21.678369 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd668c7-2729-4f3f-acef-c20a7baf76ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:21.678635 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd668c7-2729-4f3f-acef-c20a7baf76ed admin admin] PATH_INFO: `/v3/projects/7e340a45aec4486ca8a50e2f2f7e231e/users/b8c789926abe40ceb7c17b0180c07472/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:21.680733 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcd668c7-2729-4f3f-acef-c20a7baf76ed admin admin] RBAC: Authorizing `identity:create_grant(project_id=7e340a45aec4486ca8a50e2f2f7e231e, user_id=b8c789926abe40ceb7c17b0180c07472, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:21.683457 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcd668c7-2729-4f3f-acef-c20a7baf76ed admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:21.696601 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1407/2831] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 01:08:21 2022] PUT /identity/v3/projects/7e340a45aec4486ca8a50e2f2f7e231e/users/b8c789926abe40ceb7c17b0180c07472/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 01:08:21.708999 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1018a195-37cd-4aff-b2c8-6a7b4a56c38d None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:21.709547 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1018a195-37cd-4aff-b2c8-6a7b4a56c38d None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:21.710016 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1018a195-37cd-4aff-b2c8-6a7b4a56c38d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:21.746852 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-1018a195-37cd-4aff-b2c8-6a7b4a56c38d None None] MFA Rules not processed for user `b8c789926abe40ceb7c17b0180c07472`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:08:21.748343 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-1018a195-37cd-4aff-b2c8-6a7b4a56c38d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:08:21.835086 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1425/2832] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:08:21 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:08:21.852529 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd668c7-2729-4f3f-acef-c20a7baf76ed admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:21.855138 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-dcd668c7-2729-4f3f-acef-c20a7baf76ed admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:08:21.958000 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd668c7-2729-4f3f-acef-c20a7baf76ed admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:21.962058 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a418bc21-d9ff-40e5-849f-470645455cd4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:21.962898 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a418bc21-d9ff-40e5-849f-470645455cd4 service cinder] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:21.963298 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a418bc21-d9ff-40e5-849f-470645455cd4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:21.963662 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a418bc21-d9ff-40e5-849f-470645455cd4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:21.974462 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a418bc21-d9ff-40e5-849f-470645455cd4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:21.975914 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a418bc21-d9ff-40e5-849f-470645455cd4 service cinder] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:21.988223 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1408/2833] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:08:21 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:08:22.081161 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1018a195-37cd-4aff-b2c8-6a7b4a56c38d None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:22.093045 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1018a195-37cd-4aff-b2c8-6a7b4a56c38d None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:22.096947 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-051ffd1e-0d40-4ac9-9f86-1905d413116d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:22.097600 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051ffd1e-0d40-4ac9-9f86-1905d413116d service cinder] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:22.097871 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051ffd1e-0d40-4ac9-9f86-1905d413116d service cinder] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:22.098136 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051ffd1e-0d40-4ac9-9f86-1905d413116d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:22.110808 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:08:22.173501 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-051ffd1e-0d40-4ac9-9f86-1905d413116d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:22.175046 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-051ffd1e-0d40-4ac9-9f86-1905d413116d service cinder] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:22.186011 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1426/2834] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:08:22 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:08:25.133331 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a418bc21-d9ff-40e5-849f-470645455cd4 service cinder] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:25.207951 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a418bc21-d9ff-40e5-849f-470645455cd4 service cinder] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:25.213222 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-268f1da7-9796-4e22-b183-aa0606b5ddeb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:25.213911 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268f1da7-9796-4e22-b183-aa0606b5ddeb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:25.214187 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268f1da7-9796-4e22-b183-aa0606b5ddeb admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:25.214456 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268f1da7-9796-4e22-b183-aa0606b5ddeb admin admin] PATH_INFO: `/v3/users/9b5b6260e1a346da9c0c64c6f64b05c4` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:25.216336 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-268f1da7-9796-4e22-b183-aa0606b5ddeb admin admin] RBAC: Authorizing `identity:delete_user(user_id=9b5b6260e1a346da9c0c64c6f64b05c4)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:25.218328 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-268f1da7-9796-4e22-b183-aa0606b5ddeb admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:25.276546 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-268f1da7-9796-4e22-b183-aa0606b5ddeb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9b5b6260e1a346da9c0c64c6f64b05c4'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:25.287650 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-268f1da7-9796-4e22-b183-aa0606b5ddeb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9b5b6260e1a346da9c0c64c6f64b05c4'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:25.288371 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-268f1da7-9796-4e22-b183-aa0606b5ddeb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9b5b6260e1a346da9c0c64c6f64b05c4'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:25.305600 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-268f1da7-9796-4e22-b183-aa0606b5ddeb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9b5b6260e1a346da9c0c64c6f64b05c4'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:25.306587 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-268f1da7-9796-4e22-b183-aa0606b5ddeb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9b5b6260e1a346da9c0c64c6f64b05c4'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:25.317915 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=88,keystone:INSERT=3,keystone:DELETE=9 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:08:25.335622 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1409/2835] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 01:08:25 2022] DELETE /identity/v3/users/9b5b6260e1a346da9c0c64c6f64b05c4 => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 01:08:25.341115 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-431f9c9d-d6c8-49e7-b326-7e65c8ea13bd None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:25.341115 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-431f9c9d-d6c8-49e7-b326-7e65c8ea13bd None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:25.342238 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-431f9c9d-d6c8-49e7-b326-7e65c8ea13bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:25.364148 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-431f9c9d-d6c8-49e7-b326-7e65c8ea13bd None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:08:25.365439 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-431f9c9d-d6c8-49e7-b326-7e65c8ea13bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:08:25.440851 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1427/2836] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:08:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:08:25.451269 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-268f1da7-9796-4e22-b183-aa0606b5ddeb admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:25.452720 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-268f1da7-9796-4e22-b183-aa0606b5ddeb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:08:25.525030 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-268f1da7-9796-4e22-b183-aa0606b5ddeb admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:25.530438 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a261942-73d6-44f2-874d-b268abd6d44f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:25.530983 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a261942-73d6-44f2-874d-b268abd6d44f service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:25.531252 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a261942-73d6-44f2-874d-b268abd6d44f service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:25.531515 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a261942-73d6-44f2-874d-b268abd6d44f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:25.542242 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a261942-73d6-44f2-874d-b268abd6d44f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:25.544645 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a261942-73d6-44f2-874d-b268abd6d44f service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:25.556138 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1410/2837] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:08:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 01:08:25.879476 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-431f9c9d-d6c8-49e7-b326-7e65c8ea13bd None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:25.881033 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-431f9c9d-d6c8-49e7-b326-7e65c8ea13bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:08:25.896662 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-431f9c9d-d6c8-49e7-b326-7e65c8ea13bd None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:25.905031 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d9b016-2334-4db5-945c-bb72da840b0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:25.906132 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d9b016-2334-4db5-945c-bb72da840b0a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:25.906593 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d9b016-2334-4db5-945c-bb72da840b0a admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:25.907091 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d9b016-2334-4db5-945c-bb72da840b0a admin admin] PATH_INFO: `/v3/projects/02094d9dfdc245d38c7181f92f9f6b86` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:25.909228 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5d9b016-2334-4db5-945c-bb72da840b0a admin admin] RBAC: Authorizing `identity:delete_project(project_id=02094d9dfdc245d38c7181f92f9f6b86)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:25.912911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5d9b016-2334-4db5-945c-bb72da840b0a admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:25.976170 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-b5d9b016-2334-4db5-945c-bb72da840b0a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '02094d9dfdc245d38c7181f92f9f6b86', 'request_id': 'req-b5d9b016-2334-4db5-945c-bb72da840b0a'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:25.986235 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-b5d9b016-2334-4db5-945c-bb72da840b0a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '02094d9dfdc245d38c7181f92f9f6b86', 'request_id': 'req-b5d9b016-2334-4db5-945c-bb72da840b0a'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:25.993871 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-b5d9b016-2334-4db5-945c-bb72da840b0a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '02094d9dfdc245d38c7181f92f9f6b86', 'request_id': 'req-b5d9b016-2334-4db5-945c-bb72da840b0a'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:26.008083 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-b5d9b016-2334-4db5-945c-bb72da840b0a admin admin] The token cache is being invalidate because project 02094d9dfdc245d38c7181f92f9f6b86 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 01:08:26.008375 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-b5d9b016-2334-4db5-945c-bb72da840b0a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:26.009055 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-b5d9b016-2334-4db5-945c-bb72da840b0a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:26.010635 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1428/2838] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 01:08:25 2022] DELETE /identity/v3/projects/02094d9dfdc245d38c7181f92f9f6b86 => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 01:08:26.015382 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a261942-73d6-44f2-874d-b268abd6d44f service neutron] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:26.026034 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a261942-73d6-44f2-874d-b268abd6d44f service neutron] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:26.092493 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-698e79ac-b8f0-4925-8a06-240d8002d722 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:26.093044 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698e79ac-b8f0-4925-8a06-240d8002d722 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:26.093356 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698e79ac-b8f0-4925-8a06-240d8002d722 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:26.093628 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698e79ac-b8f0-4925-8a06-240d8002d722 admin admin] PATH_INFO: `/v3/users/b8c789926abe40ceb7c17b0180c07472` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:26.094867 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-698e79ac-b8f0-4925-8a06-240d8002d722 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b8c789926abe40ceb7c17b0180c07472)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:26.096913 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-698e79ac-b8f0-4925-8a06-240d8002d722 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:26.156388 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-698e79ac-b8f0-4925-8a06-240d8002d722 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b8c789926abe40ceb7c17b0180c07472'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:26.165755 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-698e79ac-b8f0-4925-8a06-240d8002d722 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b8c789926abe40ceb7c17b0180c07472'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:26.166439 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-698e79ac-b8f0-4925-8a06-240d8002d722 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b8c789926abe40ceb7c17b0180c07472'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:26.174821 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-698e79ac-b8f0-4925-8a06-240d8002d722 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b8c789926abe40ceb7c17b0180c07472'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:26.175579 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-698e79ac-b8f0-4925-8a06-240d8002d722 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b8c789926abe40ceb7c17b0180c07472'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:26.197783 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1411/2839] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 01:08:26 2022] DELETE /identity/v3/users/b8c789926abe40ceb7c17b0180c07472 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 01:08:26.533939 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d9b016-2334-4db5-945c-bb72da840b0a admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:26.535793 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-b5d9b016-2334-4db5-945c-bb72da840b0a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:08:26.551454 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d9b016-2334-4db5-945c-bb72da840b0a admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:26.597949 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:08:26.629930 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd65af4-9043-4b81-a261-ca0f1c7fb334 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:26.630507 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd65af4-9043-4b81-a261-ca0f1c7fb334 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:26.630786 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd65af4-9043-4b81-a261-ca0f1c7fb334 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:26.631056 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd65af4-9043-4b81-a261-ca0f1c7fb334 admin admin] PATH_INFO: `/v3/projects/7e340a45aec4486ca8a50e2f2f7e231e` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:26.632363 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fd65af4-9043-4b81-a261-ca0f1c7fb334 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7e340a45aec4486ca8a50e2f2f7e231e)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:26.634372 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fd65af4-9043-4b81-a261-ca0f1c7fb334 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:26.694496 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-8fd65af4-9043-4b81-a261-ca0f1c7fb334 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7e340a45aec4486ca8a50e2f2f7e231e', 'request_id': 'req-8fd65af4-9043-4b81-a261-ca0f1c7fb334'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:26.706559 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-8fd65af4-9043-4b81-a261-ca0f1c7fb334 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7e340a45aec4486ca8a50e2f2f7e231e', 'request_id': 'req-8fd65af4-9043-4b81-a261-ca0f1c7fb334'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:26.716911 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-8fd65af4-9043-4b81-a261-ca0f1c7fb334 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7e340a45aec4486ca8a50e2f2f7e231e', 'request_id': 'req-8fd65af4-9043-4b81-a261-ca0f1c7fb334'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:26.734702 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-8fd65af4-9043-4b81-a261-ca0f1c7fb334 admin admin] The token cache is being invalidate because project 7e340a45aec4486ca8a50e2f2f7e231e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 01:08:26.735032 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-8fd65af4-9043-4b81-a261-ca0f1c7fb334 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:26.735847 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-8fd65af4-9043-4b81-a261-ca0f1c7fb334 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:26.738220 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1429/2840] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 01:08:26 2022] DELETE /identity/v3/projects/7e340a45aec4486ca8a50e2f2f7e231e => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:08:26.743420 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea20ac63-3749-4fce-87dc-b109d9225a0a None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:26.743829 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea20ac63-3749-4fce-87dc-b109d9225a0a None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:26.744246 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea20ac63-3749-4fce-87dc-b109d9225a0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:26.766625 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-ea20ac63-3749-4fce-87dc-b109d9225a0a None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:08:26.767892 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-ea20ac63-3749-4fce-87dc-b109d9225a0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:08:26.774557 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:08:26.841016 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1412/2841] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:08:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:08:26.849293 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7463c05a-9017-4592-86bd-1d4b3c5044f6 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:26.849586 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7463c05a-9017-4592-86bd-1d4b3c5044f6 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:26.849850 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7463c05a-9017-4592-86bd-1d4b3c5044f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:26.872574 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-7463c05a-9017-4592-86bd-1d4b3c5044f6 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:08:26.874092 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-7463c05a-9017-4592-86bd-1d4b3c5044f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:08:26.882968 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1430/2842] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:08:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:08:26.887852 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea20ac63-3749-4fce-87dc-b109d9225a0a None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:26.899415 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea20ac63-3749-4fce-87dc-b109d9225a0a None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:26.904930 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f667b9dd-3720-443d-b737-6e03ba086396 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:26.905600 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f667b9dd-3720-443d-b737-6e03ba086396 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:26.905875 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f667b9dd-3720-443d-b737-6e03ba086396 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:26.906138 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f667b9dd-3720-443d-b737-6e03ba086396 admin admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:26.906798 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f667b9dd-3720-443d-b737-6e03ba086396 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 01:08:26.907098 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f667b9dd-3720-443d-b737-6e03ba086396 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:26.908732 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f667b9dd-3720-443d-b737-6e03ba086396 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:26.923131 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1413/2843] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 01:08:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 01:08:26.927872 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7463c05a-9017-4592-86bd-1d4b3c5044f6 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:26.939231 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7463c05a-9017-4592-86bd-1d4b3c5044f6 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:26.944320 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efab2475-85fb-4dcd-94f7-98633b7b4d1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:26.944976 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efab2475-85fb-4dcd-94f7-98633b7b4d1c admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:26.945283 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efab2475-85fb-4dcd-94f7-98633b7b4d1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:26.945560 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efab2475-85fb-4dcd-94f7-98633b7b4d1c admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:26.946227 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efab2475-85fb-4dcd-94f7-98633b7b4d1c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:26.948199 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efab2475-85fb-4dcd-94f7-98633b7b4d1c admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:26.981508 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1431/2844] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 01:08:26 2022] POST /identity/v3/projects => generated 381 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 01:08:26.988105 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f667b9dd-3720-443d-b737-6e03ba086396 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:26.999839 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f667b9dd-3720-443d-b737-6e03ba086396 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:27.064684 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52e01646-5223-4d89-8ef6-5319f31f6786 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:27.065551 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e01646-5223-4d89-8ef6-5319f31f6786 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:27.065922 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e01646-5223-4d89-8ef6-5319f31f6786 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:27.066315 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e01646-5223-4d89-8ef6-5319f31f6786 admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:27.067107 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52e01646-5223-4d89-8ef6-5319f31f6786 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:27.069192 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52e01646-5223-4d89-8ef6-5319f31f6786 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:27.093544 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1414/2845] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 01:08:26 2022] POST /identity/v3/users => generated 291 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 01:08:27.100126 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efab2475-85fb-4dcd-94f7-98633b7b4d1c admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:27.118786 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efab2475-85fb-4dcd-94f7-98633b7b4d1c admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:27.126866 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48082106-db7d-4390-aab9-fb76192b2db9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:27.127909 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48082106-db7d-4390-aab9-fb76192b2db9 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:27.128398 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48082106-db7d-4390-aab9-fb76192b2db9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:27.128868 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48082106-db7d-4390-aab9-fb76192b2db9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:27.130138 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48082106-db7d-4390-aab9-fb76192b2db9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:27.133341 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48082106-db7d-4390-aab9-fb76192b2db9 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:27.153983 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1432/2846] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:08:27 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 01:08:27.158351 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52e01646-5223-4d89-8ef6-5319f31f6786 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:27.171205 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52e01646-5223-4d89-8ef6-5319f31f6786 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:27.176305 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14a82c4-7247-4d69-a474-de874960241e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:27.176935 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14a82c4-7247-4d69-a474-de874960241e admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:27.177212 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14a82c4-7247-4d69-a474-de874960241e admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:27.177519 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14a82c4-7247-4d69-a474-de874960241e admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:27.178187 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d14a82c4-7247-4d69-a474-de874960241e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:27.179962 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d14a82c4-7247-4d69-a474-de874960241e admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:27.193460 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1415/2847] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:08:27 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 01:08:27.198112 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48082106-db7d-4390-aab9-fb76192b2db9 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:27.210410 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48082106-db7d-4390-aab9-fb76192b2db9 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:27.216158 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a507d503-c36f-4b12-ae94-a76a92896aea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:27.216829 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a507d503-c36f-4b12-ae94-a76a92896aea admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:27.217107 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a507d503-c36f-4b12-ae94-a76a92896aea admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:27.217427 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a507d503-c36f-4b12-ae94-a76a92896aea admin admin] PATH_INFO: `/v3/projects/63a1bee4947648e683d48d513aa1be5d/users/171dd04be856473785bd3f536851241f/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:27.235730 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a507d503-c36f-4b12-ae94-a76a92896aea admin admin] RBAC: Authorizing `identity:create_grant(project_id=63a1bee4947648e683d48d513aa1be5d, user_id=171dd04be856473785bd3f536851241f, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:27.238750 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a507d503-c36f-4b12-ae94-a76a92896aea admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:27.255010 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1433/2848] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 01:08:27 2022] PUT /identity/v3/projects/63a1bee4947648e683d48d513aa1be5d/users/171dd04be856473785bd3f536851241f/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:08:27.268024 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21b9e42-a898-4fa7-ae12-f7ca61cbab27 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:27.268323 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21b9e42-a898-4fa7-ae12-f7ca61cbab27 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:27.268553 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21b9e42-a898-4fa7-ae12-f7ca61cbab27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:27.302853 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-d21b9e42-a898-4fa7-ae12-f7ca61cbab27 None None] MFA Rules not processed for user `171dd04be856473785bd3f536851241f`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:08:27.304187 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-d21b9e42-a898-4fa7-ae12-f7ca61cbab27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:08:27.384149 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1416/2849] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:08:27 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:08:27.390042 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a507d503-c36f-4b12-ae94-a76a92896aea admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:27.476871 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a507d503-c36f-4b12-ae94-a76a92896aea admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:27.483300 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7bd31d0-6453-44aa-b29d-92a461e0ca7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:27.484380 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7bd31d0-6453-44aa-b29d-92a461e0ca7f admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:27.484751 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7bd31d0-6453-44aa-b29d-92a461e0ca7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:27.485226 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7bd31d0-6453-44aa-b29d-92a461e0ca7f admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:27.486647 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7bd31d0-6453-44aa-b29d-92a461e0ca7f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:27.488732 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7bd31d0-6453-44aa-b29d-92a461e0ca7f admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:27.494977 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:08:27.522864 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1434/2850] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 01:08:27 2022] POST /identity/v3/projects => generated 383 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 01:08:27.527912 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21b9e42-a898-4fa7-ae12-f7ca61cbab27 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:27.540308 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21b9e42-a898-4fa7-ae12-f7ca61cbab27 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:27.545084 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:08:27.616255 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2f7b97-e36a-48af-9cf2-efac6df11456 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:27.616881 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2f7b97-e36a-48af-9cf2-efac6df11456 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:27.617154 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2f7b97-e36a-48af-9cf2-efac6df11456 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:27.617459 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2f7b97-e36a-48af-9cf2-efac6df11456 admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:27.618103 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df2f7b97-e36a-48af-9cf2-efac6df11456 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:27.620140 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df2f7b97-e36a-48af-9cf2-efac6df11456 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:27.647137 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1417/2851] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 01:08:27 2022] POST /identity/v3/users => generated 292 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 01:08:27.651630 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7bd31d0-6453-44aa-b29d-92a461e0ca7f admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:27.663705 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7bd31d0-6453-44aa-b29d-92a461e0ca7f admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:27.669994 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d8b43f-ea5c-49d6-9345-e6f6ad4c9261 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:27.670627 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d8b43f-ea5c-49d6-9345-e6f6ad4c9261 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:27.670909 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d8b43f-ea5c-49d6-9345-e6f6ad4c9261 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:27.671183 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d8b43f-ea5c-49d6-9345-e6f6ad4c9261 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:27.671883 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32d8b43f-ea5c-49d6-9345-e6f6ad4c9261 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:27.673581 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32d8b43f-ea5c-49d6-9345-e6f6ad4c9261 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:27.686918 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1435/2852] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:08:27 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 01:08:27.691777 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2f7b97-e36a-48af-9cf2-efac6df11456 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:27.704377 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2f7b97-e36a-48af-9cf2-efac6df11456 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:27.709789 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-698cd521-e936-47cc-9e17-a7e0d61fc003 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:27.710416 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698cd521-e936-47cc-9e17-a7e0d61fc003 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:27.710696 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698cd521-e936-47cc-9e17-a7e0d61fc003 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:27.710966 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698cd521-e936-47cc-9e17-a7e0d61fc003 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:27.711629 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-698cd521-e936-47cc-9e17-a7e0d61fc003 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:27.713314 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-698cd521-e936-47cc-9e17-a7e0d61fc003 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:27.726534 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1418/2853] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:08:27 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 01:08:27.734310 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d8b43f-ea5c-49d6-9345-e6f6ad4c9261 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:27.753534 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d8b43f-ea5c-49d6-9345-e6f6ad4c9261 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:27.761630 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d557cdf1-9274-4b6f-a1f6-8d44ea0cf2c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:27.762774 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d557cdf1-9274-4b6f-a1f6-8d44ea0cf2c6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:27.763257 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d557cdf1-9274-4b6f-a1f6-8d44ea0cf2c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:27.763735 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d557cdf1-9274-4b6f-a1f6-8d44ea0cf2c6 admin admin] PATH_INFO: `/v3/projects/5a9894e9728943c79f2abff50b183d69/users/6bb57e5bd68a4ef69660ddab3074b8fe/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:27.794229 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d557cdf1-9274-4b6f-a1f6-8d44ea0cf2c6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5a9894e9728943c79f2abff50b183d69, user_id=6bb57e5bd68a4ef69660ddab3074b8fe, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:27.799748 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d557cdf1-9274-4b6f-a1f6-8d44ea0cf2c6 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:27.822977 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1436/2854] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 01:08:27 2022] PUT /identity/v3/projects/5a9894e9728943c79f2abff50b183d69/users/6bb57e5bd68a4ef69660ddab3074b8fe/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 90 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:08:27.844148 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ff5635-4011-440f-9f82-99e30d3dbff5 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:27.844549 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ff5635-4011-440f-9f82-99e30d3dbff5 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:27.844870 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ff5635-4011-440f-9f82-99e30d3dbff5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:27.877227 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-f5ff5635-4011-440f-9f82-99e30d3dbff5 None None] MFA Rules not processed for user `6bb57e5bd68a4ef69660ddab3074b8fe`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:08:27.878518 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-f5ff5635-4011-440f-9f82-99e30d3dbff5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:08:27.957500 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1419/2855] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:08:27 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:08:27.962286 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d557cdf1-9274-4b6f-a1f6-8d44ea0cf2c6 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:28.034591 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d557cdf1-9274-4b6f-a1f6-8d44ea0cf2c6 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:28.039955 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e172ef-5dc2-46b1-a80b-6fe6b5f99858 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:28.040577 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e172ef-5dc2-46b1-a80b-6fe6b5f99858 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:28.040850 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e172ef-5dc2-46b1-a80b-6fe6b5f99858 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:28.041113 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e172ef-5dc2-46b1-a80b-6fe6b5f99858 admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:28.041783 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e172ef-5dc2-46b1-a80b-6fe6b5f99858 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:28.044998 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e172ef-5dc2-46b1-a80b-6fe6b5f99858 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:28.076720 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1437/2856] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 01:08:27 2022] POST /identity/v3/projects => generated 385 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 01:08:28.081011 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ff5635-4011-440f-9f82-99e30d3dbff5 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:28.093653 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ff5635-4011-440f-9f82-99e30d3dbff5 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:28.180449 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b93cb9-2cba-42dd-bb5d-f335a2e15122 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:28.181210 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b93cb9-2cba-42dd-bb5d-f335a2e15122 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:28.181405 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b93cb9-2cba-42dd-bb5d-f335a2e15122 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:28.181634 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b93cb9-2cba-42dd-bb5d-f335a2e15122 admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:28.182295 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9b93cb9-2cba-42dd-bb5d-f335a2e15122 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:28.184541 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9b93cb9-2cba-42dd-bb5d-f335a2e15122 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:28.213028 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1420/2857] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 01:08:28 2022] POST /identity/v3/users => generated 299 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 01:08:28.217575 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e172ef-5dc2-46b1-a80b-6fe6b5f99858 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:28.229511 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e172ef-5dc2-46b1-a80b-6fe6b5f99858 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:28.235245 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b4378f9-54f1-4bcc-ba07-5d6a8b37f7b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:28.235866 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4378f9-54f1-4bcc-ba07-5d6a8b37f7b5 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:28.236143 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4378f9-54f1-4bcc-ba07-5d6a8b37f7b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:28.236418 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4378f9-54f1-4bcc-ba07-5d6a8b37f7b5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:28.237080 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b4378f9-54f1-4bcc-ba07-5d6a8b37f7b5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:28.239015 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b4378f9-54f1-4bcc-ba07-5d6a8b37f7b5 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:28.252055 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1438/2858] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:08:28 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 01:08:28.256526 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b93cb9-2cba-42dd-bb5d-f335a2e15122 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:28.264279 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:08:28.268914 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b93cb9-2cba-42dd-bb5d-f335a2e15122 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:28.277651 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cfae22b-bc56-42a1-bca1-9ba7ab521ca6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:28.278785 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cfae22b-bc56-42a1-bca1-9ba7ab521ca6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:28.279423 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cfae22b-bc56-42a1-bca1-9ba7ab521ca6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:28.279866 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cfae22b-bc56-42a1-bca1-9ba7ab521ca6 admin admin] PATH_INFO: `/v3/projects/95b69cfefdb44334be59ce7ef070079f/users/35193c0563b944d295a979aaa0e762ef/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:28.314020 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cfae22b-bc56-42a1-bca1-9ba7ab521ca6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=95b69cfefdb44334be59ce7ef070079f, user_id=35193c0563b944d295a979aaa0e762ef, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:28.321885 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cfae22b-bc56-42a1-bca1-9ba7ab521ca6 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:28.348115 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1421/2859] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 01:08:28 2022] PUT /identity/v3/projects/95b69cfefdb44334be59ce7ef070079f/users/35193c0563b944d295a979aaa0e762ef/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 92 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 01:08:28.352531 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b4378f9-54f1-4bcc-ba07-5d6a8b37f7b5 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:28.364389 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b4378f9-54f1-4bcc-ba07-5d6a8b37f7b5 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:28.433312 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cfdecff-0170-466a-8a50-2b2c476347d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:28.434562 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cfdecff-0170-466a-8a50-2b2c476347d9 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:28.434839 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cfdecff-0170-466a-8a50-2b2c476347d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:28.435380 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cfdecff-0170-466a-8a50-2b2c476347d9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:28.436096 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cfdecff-0170-466a-8a50-2b2c476347d9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:28.437942 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cfdecff-0170-466a-8a50-2b2c476347d9 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:28.449112 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1439/2860] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:08:28 2022] GET /identity/v3/roles => generated 1420 bytes in 97 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 01:08:28.454397 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cfae22b-bc56-42a1-bca1-9ba7ab521ca6 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:28.470794 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cfae22b-bc56-42a1-bca1-9ba7ab521ca6 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:28.478481 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c91cb11-3072-4576-8095-79b40e037c9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:28.479479 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c91cb11-3072-4576-8095-79b40e037c9a admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:28.479915 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c91cb11-3072-4576-8095-79b40e037c9a admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:28.480600 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c91cb11-3072-4576-8095-79b40e037c9a admin admin] PATH_INFO: `/v3/projects/95b69cfefdb44334be59ce7ef070079f/users/35193c0563b944d295a979aaa0e762ef/roles/2c242abf6ed8410ab0a4a354ca1794fb` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:28.484257 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c91cb11-3072-4576-8095-79b40e037c9a admin admin] RBAC: Authorizing `identity:create_grant(project_id=95b69cfefdb44334be59ce7ef070079f, user_id=35193c0563b944d295a979aaa0e762ef, role_id=2c242abf6ed8410ab0a4a354ca1794fb)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:28.489852 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c91cb11-3072-4576-8095-79b40e037c9a admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:28.508434 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1422/2861] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 01:08:28 2022] PUT /identity/v3/projects/95b69cfefdb44334be59ce7ef070079f/users/35193c0563b944d295a979aaa0e762ef/roles/2c242abf6ed8410ab0a4a354ca1794fb => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:08:28.519776 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef267df0-d6d0-4486-a9ef-e3a8b5758e5e None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:28.520157 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef267df0-d6d0-4486-a9ef-e3a8b5758e5e None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:28.520434 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef267df0-d6d0-4486-a9ef-e3a8b5758e5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:28.552408 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-ef267df0-d6d0-4486-a9ef-e3a8b5758e5e None None] MFA Rules not processed for user `35193c0563b944d295a979aaa0e762ef`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:08:28.553767 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-ef267df0-d6d0-4486-a9ef-e3a8b5758e5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:08:28.564818 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:08:28.665069 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1440/2862] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:08:28 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 01:08:28.678129 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c91cb11-3072-4576-8095-79b40e037c9a admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:28.680099 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-6c91cb11-3072-4576-8095-79b40e037c9a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:08:28.764993 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c91cb11-3072-4576-8095-79b40e037c9a admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:28.771308 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab380b6e-1462-4051-98e8-10c7c0b4171c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:28.772352 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab380b6e-1462-4051-98e8-10c7c0b4171c service cinder] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:28.772842 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab380b6e-1462-4051-98e8-10c7c0b4171c service cinder] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:28.773360 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab380b6e-1462-4051-98e8-10c7c0b4171c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:28.881081 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab380b6e-1462-4051-98e8-10c7c0b4171c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:28.883939 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab380b6e-1462-4051-98e8-10c7c0b4171c service cinder] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:28.902191 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1423/2863] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:08:28 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 225 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 01:08:30.521639 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef267df0-d6d0-4486-a9ef-e3a8b5758e5e None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:30.523128 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-ef267df0-d6d0-4486-a9ef-e3a8b5758e5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:08:30.536427 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef267df0-d6d0-4486-a9ef-e3a8b5758e5e None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:30.540349 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35ac7f2-47bb-4090-a2f0-7d52981c7798 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:30.540977 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35ac7f2-47bb-4090-a2f0-7d52981c7798 service cinder] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:30.541274 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35ac7f2-47bb-4090-a2f0-7d52981c7798 service cinder] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:30.541562 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35ac7f2-47bb-4090-a2f0-7d52981c7798 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:30.606710 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b35ac7f2-47bb-4090-a2f0-7d52981c7798 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:30.608091 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b35ac7f2-47bb-4090-a2f0-7d52981c7798 service cinder] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:30.618818 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1441/2864] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:08:30 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:08:31.244331 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab380b6e-1462-4051-98e8-10c7c0b4171c service cinder] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:31.254371 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab380b6e-1462-4051-98e8-10c7c0b4171c service cinder] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:31.258116 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-488994de-2619-43cb-8996-a31e44c69942 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:31.258730 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488994de-2619-43cb-8996-a31e44c69942 service cinder] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:31.259002 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488994de-2619-43cb-8996-a31e44c69942 service cinder] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:31.259267 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488994de-2619-43cb-8996-a31e44c69942 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:31.271693 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-488994de-2619-43cb-8996-a31e44c69942 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:31.274019 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-488994de-2619-43cb-8996-a31e44c69942 service cinder] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:31.287445 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1424/2865] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:08:31 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:08:35.409316 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35ac7f2-47bb-4090-a2f0-7d52981c7798 service cinder] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:35.485096 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35ac7f2-47bb-4090-a2f0-7d52981c7798 service cinder] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:35.490377 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead652c7-e4c9-406b-9eb9-f4f62f4ead91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:35.491063 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead652c7-e4c9-406b-9eb9-f4f62f4ead91 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:35.491379 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead652c7-e4c9-406b-9eb9-f4f62f4ead91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:35.491587 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead652c7-e4c9-406b-9eb9-f4f62f4ead91 admin admin] PATH_INFO: `/v3/users/171dd04be856473785bd3f536851241f` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:35.492785 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ead652c7-e4c9-406b-9eb9-f4f62f4ead91 admin admin] RBAC: Authorizing `identity:delete_user(user_id=171dd04be856473785bd3f536851241f)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:35.494832 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ead652c7-e4c9-406b-9eb9-f4f62f4ead91 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:35.565019 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-ead652c7-e4c9-406b-9eb9-f4f62f4ead91 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '171dd04be856473785bd3f536851241f'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:35.565890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-ead652c7-e4c9-406b-9eb9-f4f62f4ead91 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '171dd04be856473785bd3f536851241f'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:35.568157 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:08:35.577832 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-ead652c7-e4c9-406b-9eb9-f4f62f4ead91 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '171dd04be856473785bd3f536851241f'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:35.578536 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-ead652c7-e4c9-406b-9eb9-f4f62f4ead91 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '171dd04be856473785bd3f536851241f'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:35.587043 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-ead652c7-e4c9-406b-9eb9-f4f62f4ead91 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '171dd04be856473785bd3f536851241f'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:35.607566 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1442/2866] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 01:08:35 2022] DELETE /identity/v3/users/171dd04be856473785bd3f536851241f => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 01:08:35.612279 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286a0a63-3118-4c1e-a730-5334a1ef5e0e None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:35.613239 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286a0a63-3118-4c1e-a730-5334a1ef5e0e None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:35.613239 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286a0a63-3118-4c1e-a730-5334a1ef5e0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:35.634452 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-286a0a63-3118-4c1e-a730-5334a1ef5e0e None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:08:35.635934 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-286a0a63-3118-4c1e-a730-5334a1ef5e0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:08:35.712113 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1425/2867] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:08:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:08:35.723632 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead652c7-e4c9-406b-9eb9-f4f62f4ead91 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:35.725384 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-ead652c7-e4c9-406b-9eb9-f4f62f4ead91 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:08:35.794247 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead652c7-e4c9-406b-9eb9-f4f62f4ead91 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:35.798237 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a6ad682-dc15-4f6d-b01c-a8502f1c8773 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:35.798856 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6ad682-dc15-4f6d-b01c-a8502f1c8773 service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:35.799135 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6ad682-dc15-4f6d-b01c-a8502f1c8773 service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:35.799407 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6ad682-dc15-4f6d-b01c-a8502f1c8773 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:35.811282 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a6ad682-dc15-4f6d-b01c-a8502f1c8773 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:35.813059 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a6ad682-dc15-4f6d-b01c-a8502f1c8773 service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:35.825614 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1443/2868] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:08:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:08:36.232681 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286a0a63-3118-4c1e-a730-5334a1ef5e0e None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:36.234687 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-286a0a63-3118-4c1e-a730-5334a1ef5e0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:08:36.254845 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286a0a63-3118-4c1e-a730-5334a1ef5e0e None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:36.263741 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13434d9d-431c-438a-8df4-5988a4f828df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:36.265048 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13434d9d-431c-438a-8df4-5988a4f828df admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:36.265488 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13434d9d-431c-438a-8df4-5988a4f828df admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:36.265825 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13434d9d-431c-438a-8df4-5988a4f828df admin admin] PATH_INFO: `/v3/projects/63a1bee4947648e683d48d513aa1be5d` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:36.267392 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13434d9d-431c-438a-8df4-5988a4f828df admin admin] RBAC: Authorizing `identity:delete_project(project_id=63a1bee4947648e683d48d513aa1be5d)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:36.270617 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13434d9d-431c-438a-8df4-5988a4f828df admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:36.275836 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:08:36.334428 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-13434d9d-431c-438a-8df4-5988a4f828df admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '63a1bee4947648e683d48d513aa1be5d', 'request_id': 'req-13434d9d-431c-438a-8df4-5988a4f828df'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:36.347531 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-13434d9d-431c-438a-8df4-5988a4f828df admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '63a1bee4947648e683d48d513aa1be5d', 'request_id': 'req-13434d9d-431c-438a-8df4-5988a4f828df'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:36.354746 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-13434d9d-431c-438a-8df4-5988a4f828df admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '63a1bee4947648e683d48d513aa1be5d', 'request_id': 'req-13434d9d-431c-438a-8df4-5988a4f828df'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:36.369164 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-13434d9d-431c-438a-8df4-5988a4f828df admin admin] The token cache is being invalidate because project 63a1bee4947648e683d48d513aa1be5d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81386) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 01:08:36.369495 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-13434d9d-431c-438a-8df4-5988a4f828df admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:36.370168 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-13434d9d-431c-438a-8df4-5988a4f828df admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:36.372209 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1426/2869] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 01:08:36 2022] DELETE /identity/v3/projects/63a1bee4947648e683d48d513aa1be5d => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 01:08:36.377170 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a6ad682-dc15-4f6d-b01c-a8502f1c8773 service neutron] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:36.388758 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a6ad682-dc15-4f6d-b01c-a8502f1c8773 service neutron] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:36.461648 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe9e172-fd58-4511-af70-65c688d07646 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:36.462368 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe9e172-fd58-4511-af70-65c688d07646 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:36.462566 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe9e172-fd58-4511-af70-65c688d07646 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:36.464950 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe9e172-fd58-4511-af70-65c688d07646 admin admin] PATH_INFO: `/v3/users/6bb57e5bd68a4ef69660ddab3074b8fe` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:36.464950 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfe9e172-fd58-4511-af70-65c688d07646 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6bb57e5bd68a4ef69660ddab3074b8fe)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:36.467440 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfe9e172-fd58-4511-af70-65c688d07646 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:36.530396 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-dfe9e172-fd58-4511-af70-65c688d07646 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6bb57e5bd68a4ef69660ddab3074b8fe'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:36.531091 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-dfe9e172-fd58-4511-af70-65c688d07646 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6bb57e5bd68a4ef69660ddab3074b8fe'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:36.539985 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-dfe9e172-fd58-4511-af70-65c688d07646 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6bb57e5bd68a4ef69660ddab3074b8fe'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:36.540670 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-dfe9e172-fd58-4511-af70-65c688d07646 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6bb57e5bd68a4ef69660ddab3074b8fe'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:36.549968 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-dfe9e172-fd58-4511-af70-65c688d07646 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6bb57e5bd68a4ef69660ddab3074b8fe'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:36.556664 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:INSERT=2,keystone:DELETE=12,keystone:SELECT=86 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:08:36.578488 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1444/2870] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 01:08:36 2022] DELETE /identity/v3/users/6bb57e5bd68a4ef69660ddab3074b8fe => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:08:37.309083 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13434d9d-431c-438a-8df4-5988a4f828df admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:37.310944 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-13434d9d-431c-438a-8df4-5988a4f828df admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:08:37.333385 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13434d9d-431c-438a-8df4-5988a4f828df admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:37.436307 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-697a5b2b-7671-47d7-9f3c-03a315896cec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:37.437458 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-697a5b2b-7671-47d7-9f3c-03a315896cec admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:37.437931 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-697a5b2b-7671-47d7-9f3c-03a315896cec admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:37.438247 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-697a5b2b-7671-47d7-9f3c-03a315896cec admin admin] PATH_INFO: `/v3/projects/5a9894e9728943c79f2abff50b183d69` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:37.439604 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-697a5b2b-7671-47d7-9f3c-03a315896cec admin admin] RBAC: Authorizing `identity:delete_project(project_id=5a9894e9728943c79f2abff50b183d69)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:37.441695 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-697a5b2b-7671-47d7-9f3c-03a315896cec admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:37.512917 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-697a5b2b-7671-47d7-9f3c-03a315896cec admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5a9894e9728943c79f2abff50b183d69', 'request_id': 'req-697a5b2b-7671-47d7-9f3c-03a315896cec'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:37.531474 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-697a5b2b-7671-47d7-9f3c-03a315896cec admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5a9894e9728943c79f2abff50b183d69', 'request_id': 'req-697a5b2b-7671-47d7-9f3c-03a315896cec'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:37.539739 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-697a5b2b-7671-47d7-9f3c-03a315896cec admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5a9894e9728943c79f2abff50b183d69', 'request_id': 'req-697a5b2b-7671-47d7-9f3c-03a315896cec'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:37.557823 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-697a5b2b-7671-47d7-9f3c-03a315896cec admin admin] The token cache is being invalidate because project 5a9894e9728943c79f2abff50b183d69 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81386) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 01:08:37.560840 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-697a5b2b-7671-47d7-9f3c-03a315896cec admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:37.560840 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-697a5b2b-7671-47d7-9f3c-03a315896cec admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:37.562598 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1427/2871] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 01:08:37 2022] DELETE /identity/v3/projects/5a9894e9728943c79f2abff50b183d69 => generated 0 bytes in 257 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 01:08:37.573243 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe9e172-fd58-4511-af70-65c688d07646 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:37.575820 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-dfe9e172-fd58-4511-af70-65c688d07646 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:08:37.592778 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe9e172-fd58-4511-af70-65c688d07646 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:37.683077 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845cdb54-d2b7-4e09-90d3-7c735b7bcac9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:37.684178 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845cdb54-d2b7-4e09-90d3-7c735b7bcac9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:37.684670 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845cdb54-d2b7-4e09-90d3-7c735b7bcac9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:37.685147 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845cdb54-d2b7-4e09-90d3-7c735b7bcac9 admin admin] PATH_INFO: `/v3/users/35193c0563b944d295a979aaa0e762ef` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:37.687533 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-845cdb54-d2b7-4e09-90d3-7c735b7bcac9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=35193c0563b944d295a979aaa0e762ef)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:37.691081 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-845cdb54-d2b7-4e09-90d3-7c735b7bcac9 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:37.762210 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-845cdb54-d2b7-4e09-90d3-7c735b7bcac9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '35193c0563b944d295a979aaa0e762ef'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:37.763034 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-845cdb54-d2b7-4e09-90d3-7c735b7bcac9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '35193c0563b944d295a979aaa0e762ef'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:37.773154 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-845cdb54-d2b7-4e09-90d3-7c735b7bcac9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '35193c0563b944d295a979aaa0e762ef'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:37.774033 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-845cdb54-d2b7-4e09-90d3-7c735b7bcac9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '35193c0563b944d295a979aaa0e762ef'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:37.785515 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-845cdb54-d2b7-4e09-90d3-7c735b7bcac9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '35193c0563b944d295a979aaa0e762ef'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:37.808266 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1445/2872] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 01:08:37 2022] DELETE /identity/v3/users/35193c0563b944d295a979aaa0e762ef => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:08:38.170000 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-697a5b2b-7671-47d7-9f3c-03a315896cec admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:38.172739 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-697a5b2b-7671-47d7-9f3c-03a315896cec admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:08:38.200922 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-697a5b2b-7671-47d7-9f3c-03a315896cec admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:38.230869 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:08:38.291162 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79fd42ce-3807-4f59-ba27-c20978ed86eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:38.291737 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79fd42ce-3807-4f59-ba27-c20978ed86eb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:38.292007 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79fd42ce-3807-4f59-ba27-c20978ed86eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:38.292270 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79fd42ce-3807-4f59-ba27-c20978ed86eb admin admin] PATH_INFO: `/v3/projects/95b69cfefdb44334be59ce7ef070079f` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:38.293726 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79fd42ce-3807-4f59-ba27-c20978ed86eb admin admin] RBAC: Authorizing `identity:delete_project(project_id=95b69cfefdb44334be59ce7ef070079f)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:38.295949 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79fd42ce-3807-4f59-ba27-c20978ed86eb admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:38.353867 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-79fd42ce-3807-4f59-ba27-c20978ed86eb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '95b69cfefdb44334be59ce7ef070079f', 'request_id': 'req-79fd42ce-3807-4f59-ba27-c20978ed86eb'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:38.364776 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-79fd42ce-3807-4f59-ba27-c20978ed86eb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '95b69cfefdb44334be59ce7ef070079f', 'request_id': 'req-79fd42ce-3807-4f59-ba27-c20978ed86eb'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:38.374285 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-79fd42ce-3807-4f59-ba27-c20978ed86eb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '95b69cfefdb44334be59ce7ef070079f', 'request_id': 'req-79fd42ce-3807-4f59-ba27-c20978ed86eb'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:38.388926 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-79fd42ce-3807-4f59-ba27-c20978ed86eb admin admin] The token cache is being invalidate because project 95b69cfefdb44334be59ce7ef070079f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81386) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 01:08:38.389340 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-79fd42ce-3807-4f59-ba27-c20978ed86eb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:38.390752 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-79fd42ce-3807-4f59-ba27-c20978ed86eb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:08:38.392932 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1428/2873] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 01:08:38 2022] DELETE /identity/v3/projects/95b69cfefdb44334be59ce7ef070079f => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:08:38.399300 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a0a815-570d-4de8-8d88-cade2b3243c9 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:38.399629 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a0a815-570d-4de8-8d88-cade2b3243c9 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:38.399967 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a0a815-570d-4de8-8d88-cade2b3243c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:38.422234 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-15a0a815-570d-4de8-8d88-cade2b3243c9 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:08:38.423608 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-15a0a815-570d-4de8-8d88-cade2b3243c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:08:38.505073 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1446/2874] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:08:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:08:38.515186 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c94fbe-f0b7-42b0-b810-87e43a830ff1 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:38.515560 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c94fbe-f0b7-42b0-b810-87e43a830ff1 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:38.515916 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c94fbe-f0b7-42b0-b810-87e43a830ff1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:38.540049 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-47c94fbe-f0b7-42b0-b810-87e43a830ff1 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:08:38.541513 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-47c94fbe-f0b7-42b0-b810-87e43a830ff1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:08:38.560087 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1429/2875] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:08:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 01:08:38.567284 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a0a815-570d-4de8-8d88-cade2b3243c9 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:38.584688 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a0a815-570d-4de8-8d88-cade2b3243c9 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:38.592328 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-998d6004-0d00-437d-9813-48b6165aa59b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:38.593408 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-998d6004-0d00-437d-9813-48b6165aa59b admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:38.593900 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-998d6004-0d00-437d-9813-48b6165aa59b admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:38.594341 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-998d6004-0d00-437d-9813-48b6165aa59b admin admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:38.595498 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-998d6004-0d00-437d-9813-48b6165aa59b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 01:08:38.595942 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-998d6004-0d00-437d-9813-48b6165aa59b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:38.598474 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-998d6004-0d00-437d-9813-48b6165aa59b admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:38.600829 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:08:38.620499 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1447/2876] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 01:08:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 01:08:38.625991 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c94fbe-f0b7-42b0-b810-87e43a830ff1 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:38.638858 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c94fbe-f0b7-42b0-b810-87e43a830ff1 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:38.644553 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e9a279b-c21c-492f-b839-3c966e082cb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:38.645380 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9a279b-c21c-492f-b839-3c966e082cb4 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:38.645666 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9a279b-c21c-492f-b839-3c966e082cb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:38.645939 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9a279b-c21c-492f-b839-3c966e082cb4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:38.646611 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e9a279b-c21c-492f-b839-3c966e082cb4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:38.648612 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e9a279b-c21c-492f-b839-3c966e082cb4 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:38.696071 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1430/2877] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 01:08:38 2022] POST /identity/v3/projects => generated 373 bytes in 71 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 01:08:38.701442 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-998d6004-0d00-437d-9813-48b6165aa59b admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:38.713198 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-998d6004-0d00-437d-9813-48b6165aa59b admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:38.793310 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0d401df-bf50-4c42-bcaa-abb95a66fcc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:38.794271 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0d401df-bf50-4c42-bcaa-abb95a66fcc0 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:38.794754 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0d401df-bf50-4c42-bcaa-abb95a66fcc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:38.795211 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0d401df-bf50-4c42-bcaa-abb95a66fcc0 admin admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:38.796302 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0d401df-bf50-4c42-bcaa-abb95a66fcc0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:38.799830 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0d401df-bf50-4c42-bcaa-abb95a66fcc0 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:38.831405 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1448/2878] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 01:08:38 2022] POST /identity/v3/users => generated 287 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 01:08:38.836080 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e9a279b-c21c-492f-b839-3c966e082cb4 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:38.847968 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e9a279b-c21c-492f-b839-3c966e082cb4 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:38.853546 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110d7b63-f0af-41d9-ae23-d686c2a94449 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:38.854261 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110d7b63-f0af-41d9-ae23-d686c2a94449 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:38.854490 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110d7b63-f0af-41d9-ae23-d686c2a94449 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:38.854817 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110d7b63-f0af-41d9-ae23-d686c2a94449 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:38.855513 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-110d7b63-f0af-41d9-ae23-d686c2a94449 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:38.857196 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-110d7b63-f0af-41d9-ae23-d686c2a94449 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:38.869164 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1431/2879] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:08:38 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 01:08:38.877097 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0d401df-bf50-4c42-bcaa-abb95a66fcc0 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:38.896177 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0d401df-bf50-4c42-bcaa-abb95a66fcc0 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:38.906305 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33be73d-63fa-4538-80b3-9c0246ea3eb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:38.907338 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33be73d-63fa-4538-80b3-9c0246ea3eb6 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:38.907804 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33be73d-63fa-4538-80b3-9c0246ea3eb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:38.908276 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33be73d-63fa-4538-80b3-9c0246ea3eb6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:38.909474 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33be73d-63fa-4538-80b3-9c0246ea3eb6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:38.912684 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33be73d-63fa-4538-80b3-9c0246ea3eb6 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:38.935619 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1449/2880] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:08:38 2022] GET /identity/v3/roles => generated 1420 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 01:08:38.940305 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110d7b63-f0af-41d9-ae23-d686c2a94449 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:38.952112 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110d7b63-f0af-41d9-ae23-d686c2a94449 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:38.957207 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723aa1e8-c2ef-4a97-af1a-ebdc7631d201 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:38.957923 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723aa1e8-c2ef-4a97-af1a-ebdc7631d201 admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:38.958202 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723aa1e8-c2ef-4a97-af1a-ebdc7631d201 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:38.958475 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723aa1e8-c2ef-4a97-af1a-ebdc7631d201 admin admin] PATH_INFO: `/v3/projects/da4aa8615ed349f684f1ea9074a4bf6e/users/7b14962aa90e4c0caceb538b43ebd2d8/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:38.976779 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-723aa1e8-c2ef-4a97-af1a-ebdc7631d201 admin admin] RBAC: Authorizing `identity:create_grant(project_id=da4aa8615ed349f684f1ea9074a4bf6e, user_id=7b14962aa90e4c0caceb538b43ebd2d8, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:38.979916 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-723aa1e8-c2ef-4a97-af1a-ebdc7631d201 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:38.996639 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1432/2881] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 01:08:38 2022] PUT /identity/v3/projects/da4aa8615ed349f684f1ea9074a4bf6e/users/7b14962aa90e4c0caceb538b43ebd2d8/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:08:39.002668 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762f625d-5f20-4a09-a847-d4fc9ec8ab66 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:39.003112 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762f625d-5f20-4a09-a847-d4fc9ec8ab66 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:39.003518 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762f625d-5f20-4a09-a847-d4fc9ec8ab66 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:39.024994 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-762f625d-5f20-4a09-a847-d4fc9ec8ab66 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:08:39.026300 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-762f625d-5f20-4a09-a847-d4fc9ec8ab66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:08:39.107679 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1450/2882] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:08:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:08:39.119204 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723aa1e8-c2ef-4a97-af1a-ebdc7631d201 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:39.121325 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-723aa1e8-c2ef-4a97-af1a-ebdc7631d201 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:08:39.200850 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:08:39.205850 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723aa1e8-c2ef-4a97-af1a-ebdc7631d201 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:39.210709 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76997073-0c19-48f7-8fb8-daf14be8b93a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:39.211476 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76997073-0c19-48f7-8fb8-daf14be8b93a service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:39.212016 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76997073-0c19-48f7-8fb8-daf14be8b93a service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:39.212807 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76997073-0c19-48f7-8fb8-daf14be8b93a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:39.225331 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76997073-0c19-48f7-8fb8-daf14be8b93a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:39.226828 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76997073-0c19-48f7-8fb8-daf14be8b93a service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:39.239365 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1433/2883] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:08:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:08:40.136962 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8312a010-a35a-4035-8dc4-d1a202af8146 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:40.137940 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8312a010-a35a-4035-8dc4-d1a202af8146 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:40.137940 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8312a010-a35a-4035-8dc4-d1a202af8146 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:40.174437 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-8312a010-a35a-4035-8dc4-d1a202af8146 None None] MFA Rules not processed for user `7b14962aa90e4c0caceb538b43ebd2d8`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:08:40.175582 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-8312a010-a35a-4035-8dc4-d1a202af8146 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:08:40.281001 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1451/2884] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:08:40 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:08:40.292402 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76997073-0c19-48f7-8fb8-daf14be8b93a service neutron] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:40.294215 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-76997073-0c19-48f7-8fb8-daf14be8b93a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:08:40.362579 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76997073-0c19-48f7-8fb8-daf14be8b93a service neutron] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:40.366702 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7390489-27ae-4cce-a093-848b5f2a59f5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:40.367331 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7390489-27ae-4cce-a093-848b5f2a59f5 service cinder] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:40.367608 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7390489-27ae-4cce-a093-848b5f2a59f5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:40.367872 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7390489-27ae-4cce-a093-848b5f2a59f5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:40.378404 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7390489-27ae-4cce-a093-848b5f2a59f5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:40.380056 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7390489-27ae-4cce-a093-848b5f2a59f5 service cinder] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:40.390912 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1434/2885] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:08:40 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:08:41.889368 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1f14cb-4db2-4ce2-8756-6cde202fd529 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:41.890243 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1f14cb-4db2-4ce2-8756-6cde202fd529 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:41.890243 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1f14cb-4db2-4ce2-8756-6cde202fd529 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:41.897994 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:08:41.921046 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-0a1f14cb-4db2-4ce2-8756-6cde202fd529 None None] MFA Rules not processed for user `7b14962aa90e4c0caceb538b43ebd2d8`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:08:41.923172 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-0a1f14cb-4db2-4ce2-8756-6cde202fd529 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:08:41.937187 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1452/2886] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:08:41 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 01:08:42.241979 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7390489-27ae-4cce-a093-848b5f2a59f5 service cinder] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:08:42.252808 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7390489-27ae-4cce-a093-848b5f2a59f5 service cinder] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:08:42.256917 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aef7708-3231-4551-b12c-0666dc018398 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:08:42.257573 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aef7708-3231-4551-b12c-0666dc018398 service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:42.257850 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aef7708-3231-4551-b12c-0666dc018398 service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:42.258125 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aef7708-3231-4551-b12c-0666dc018398 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:42.260202 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-0aef7708-3231-4551-b12c-0666dc018398 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:08:42.272620 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aef7708-3231-4551-b12c-0666dc018398 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:08:42.274369 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aef7708-3231-4551-b12c-0666dc018398 service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:08:42.284292 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1435/2887] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:08:42 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 01:08:51.912623 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=2 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:08:52.279802 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=36 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:08:57.068691 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4026a1c-5e36-41c1-8b4e-2814d0c47be7 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:08:57.069315 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4026a1c-5e36-41c1-8b4e-2814d0c47be7 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:08:57.069643 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4026a1c-5e36-41c1-8b4e-2814d0c47be7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:08:57.099737 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-d4026a1c-5e36-41c1-8b4e-2814d0c47be7 None None] MFA Rules not processed for user `7b14962aa90e4c0caceb538b43ebd2d8`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:08:57.101828 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-d4026a1c-5e36-41c1-8b4e-2814d0c47be7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:08:57.113787 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1453/2888] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:08:57 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:09:07.091608 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=3 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:09:12.105414 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aef7708-3231-4551-b12c-0666dc018398 service neutron] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:09:12.107111 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-0aef7708-3231-4551-b12c-0666dc018398 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:09:12.191923 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aef7708-3231-4551-b12c-0666dc018398 service neutron] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:09:12.197982 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d35ff48-6ff6-4c5f-bec4-4586ca6664c5 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '11a80b55dcd444df901c8bf80f2a2c26', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:09:12.198894 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d35ff48-6ff6-4c5f-bec4-4586ca6664c5 service placement] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:09:12.199184 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d35ff48-6ff6-4c5f-bec4-4586ca6664c5 service placement] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:09:12.199444 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d35ff48-6ff6-4c5f-bec4-4586ca6664c5 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:09:12.201507 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-8d35ff48-6ff6-4c5f-bec4-4586ca6664c5 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:09:12.215504 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d35ff48-6ff6-4c5f-bec4-4586ca6664c5 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:09:12.217154 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d35ff48-6ff6-4c5f-bec4-4586ca6664c5 service placement] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:09:12.229107 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1436/2889] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:09:12 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:09:18.011958 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e9656f6-b9e0-49f4-90fe-68668ee917ba None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:09:18.012561 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e9656f6-b9e0-49f4-90fe-68668ee917ba None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:09:18.012561 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e9656f6-b9e0-49f4-90fe-68668ee917ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:09:18.081138 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-9e9656f6-b9e0-49f4-90fe-68668ee917ba None None] MFA Rules not processed for user `3f20ffd8df994c1db1b1e7248db5b5a3`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:09:18.082510 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-9e9656f6-b9e0-49f4-90fe-68668ee917ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:09:18.172735 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1454/2890] 10.222.0.52 () {58 vars in 1009 bytes} [Sun Sep 25 01:09:18 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:09:18.238674 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d35ff48-6ff6-4c5f-bec4-4586ca6664c5 service placement] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:09:18.240336 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-8d35ff48-6ff6-4c5f-bec4-4586ca6664c5 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:09:18.342620 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d35ff48-6ff6-4c5f-bec4-4586ca6664c5 service placement] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:09:18.348864 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3319a62-b1b8-4649-9048-13d94b53dbd8 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da606b3906dd4171bd76fd281e03e258', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:09:18.349658 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3319a62-b1b8-4649-9048-13d94b53dbd8 service swift] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:09:18.350030 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3319a62-b1b8-4649-9048-13d94b53dbd8 service swift] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:09:18.350408 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3319a62-b1b8-4649-9048-13d94b53dbd8 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:09:18.360725 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3319a62-b1b8-4649-9048-13d94b53dbd8 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:09:18.362433 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3319a62-b1b8-4649-9048-13d94b53dbd8 service swift] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:09:18.372610 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1437/2891] 10.222.0.52 () {60 vars in 1384 bytes} [Sun Sep 25 01:09:18 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 135 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 25 01:09:28.169518 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=44 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:09:28.368527 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=73 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:09:35.880607 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198b13b6-d2be-4622-8b15-994cef6aae1a None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:09:35.881228 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198b13b6-d2be-4622-8b15-994cef6aae1a None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:09:35.881228 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198b13b6-d2be-4622-8b15-994cef6aae1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:09:35.903488 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-198b13b6-d2be-4622-8b15-994cef6aae1a None None] MFA Rules not processed for user `3f20ffd8df994c1db1b1e7248db5b5a3`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:09:35.904733 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-198b13b6-d2be-4622-8b15-994cef6aae1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:09:35.912478 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1455/2892] 10.222.0.52 () {58 vars in 967 bytes} [Sun Sep 25 01:09:35 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:09:35.926281 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3319a62-b1b8-4649-9048-13d94b53dbd8 service swift] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:09:35.938205 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3319a62-b1b8-4649-9048-13d94b53dbd8 service swift] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:09:35.944247 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0fb8da1-5be6-4bef-9add-cfde2bc5382c service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da606b3906dd4171bd76fd281e03e258', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:09:35.944980 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0fb8da1-5be6-4bef-9add-cfde2bc5382c service swift] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:09:35.945304 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0fb8da1-5be6-4bef-9add-cfde2bc5382c service swift] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:09:35.945590 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0fb8da1-5be6-4bef-9add-cfde2bc5382c service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:09:35.955581 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0fb8da1-5be6-4bef-9add-cfde2bc5382c service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:09:35.957183 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0fb8da1-5be6-4bef-9add-cfde2bc5382c service swift] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:09:35.968478 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1438/2893] 10.222.0.52 () {60 vars in 1384 bytes} [Sun Sep 25 01:09:35 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 25 01:09:38.038504 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-198b13b6-d2be-4622-8b15-994cef6aae1a None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:09:38.040334 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-198b13b6-d2be-4622-8b15-994cef6aae1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:09:38.067875 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-198b13b6-d2be-4622-8b15-994cef6aae1a None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:09:38.076871 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aacce69-7a65-4f76-babf-895ccbe30fb8 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '11a80b55dcd444df901c8bf80f2a2c26', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:09:38.077953 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aacce69-7a65-4f76-babf-895ccbe30fb8 service placement] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:09:38.078450 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aacce69-7a65-4f76-babf-895ccbe30fb8 service placement] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:09:38.078930 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aacce69-7a65-4f76-babf-895ccbe30fb8 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:09:38.082137 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-4aacce69-7a65-4f76-babf-895ccbe30fb8 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:09:38.104864 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aacce69-7a65-4f76-babf-895ccbe30fb8 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:09:38.107868 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aacce69-7a65-4f76-babf-895ccbe30fb8 service placement] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:09:38.127981 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1456/2894] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:09:38 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:09:38.171040 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0fb8da1-5be6-4bef-9add-cfde2bc5382c service swift] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:09:38.183393 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0fb8da1-5be6-4bef-9add-cfde2bc5382c service swift] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:09:38.188665 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-049b09c1-72f8-4fb0-a290-751f136f1462 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:09:38.189851 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049b09c1-72f8-4fb0-a290-751f136f1462 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:09:38.190399 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049b09c1-72f8-4fb0-a290-751f136f1462 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:09:38.190576 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049b09c1-72f8-4fb0-a290-751f136f1462 admin admin] PATH_INFO: `/v3/users/7b14962aa90e4c0caceb538b43ebd2d8` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:09:38.191898 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-049b09c1-72f8-4fb0-a290-751f136f1462 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7b14962aa90e4c0caceb538b43ebd2d8)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:09:38.194071 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-049b09c1-72f8-4fb0-a290-751f136f1462 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:09:38.261931 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-049b09c1-72f8-4fb0-a290-751f136f1462 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7b14962aa90e4c0caceb538b43ebd2d8'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:09:38.270641 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-049b09c1-72f8-4fb0-a290-751f136f1462 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7b14962aa90e4c0caceb538b43ebd2d8'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:09:38.271341 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-049b09c1-72f8-4fb0-a290-751f136f1462 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7b14962aa90e4c0caceb538b43ebd2d8'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:09:38.280505 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-049b09c1-72f8-4fb0-a290-751f136f1462 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7b14962aa90e4c0caceb538b43ebd2d8'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:09:38.281570 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-049b09c1-72f8-4fb0-a290-751f136f1462 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7b14962aa90e4c0caceb538b43ebd2d8'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:09:38.304750 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1439/2895] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 01:09:38 2022] DELETE /identity/v3/users/7b14962aa90e4c0caceb538b43ebd2d8 => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 01:09:38.530627 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aacce69-7a65-4f76-babf-895ccbe30fb8 service placement] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:09:38.552331 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aacce69-7a65-4f76-babf-895ccbe30fb8 service placement] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:09:38.628203 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac8711d-41f4-49b5-ade5-ac9a2a969f58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:09:38.628890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac8711d-41f4-49b5-ade5-ac9a2a969f58 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:09:38.629204 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac8711d-41f4-49b5-ade5-ac9a2a969f58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:09:38.629563 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac8711d-41f4-49b5-ade5-ac9a2a969f58 admin admin] PATH_INFO: `/v3/projects/da4aa8615ed349f684f1ea9074a4bf6e` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:09:38.630820 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ac8711d-41f4-49b5-ade5-ac9a2a969f58 admin admin] RBAC: Authorizing `identity:delete_project(project_id=da4aa8615ed349f684f1ea9074a4bf6e)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:09:38.633131 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ac8711d-41f4-49b5-ade5-ac9a2a969f58 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:09:38.692995 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-0ac8711d-41f4-49b5-ade5-ac9a2a969f58 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'da4aa8615ed349f684f1ea9074a4bf6e', 'request_id': 'req-0ac8711d-41f4-49b5-ade5-ac9a2a969f58'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:09:38.703314 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-0ac8711d-41f4-49b5-ade5-ac9a2a969f58 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'da4aa8615ed349f684f1ea9074a4bf6e', 'request_id': 'req-0ac8711d-41f4-49b5-ade5-ac9a2a969f58'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:09:38.713908 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-0ac8711d-41f4-49b5-ade5-ac9a2a969f58 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'da4aa8615ed349f684f1ea9074a4bf6e', 'request_id': 'req-0ac8711d-41f4-49b5-ade5-ac9a2a969f58'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:09:38.736255 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-0ac8711d-41f4-49b5-ade5-ac9a2a969f58 admin admin] The token cache is being invalidate because project da4aa8615ed349f684f1ea9074a4bf6e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81387) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 01:09:38.736549 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-0ac8711d-41f4-49b5-ade5-ac9a2a969f58 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:09:38.737486 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-0ac8711d-41f4-49b5-ade5-ac9a2a969f58 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:09:38.739308 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1457/2896] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 01:09:38 2022] DELETE /identity/v3/projects/da4aa8615ed349f684f1ea9074a4bf6e => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:09:38.746887 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72006a9-01f8-492c-8ed9-2428f17012b6 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:09:38.747521 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72006a9-01f8-492c-8ed9-2428f17012b6 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:09:38.747996 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72006a9-01f8-492c-8ed9-2428f17012b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:09:38.772481 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-c72006a9-01f8-492c-8ed9-2428f17012b6 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:09:38.773678 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-c72006a9-01f8-492c-8ed9-2428f17012b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:09:38.843268 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1440/2897] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:09:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:09:38.851572 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aef510e8-70df-45d5-b86b-8392a8bd8791 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:09:38.851899 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aef510e8-70df-45d5-b86b-8392a8bd8791 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:09:38.852219 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aef510e8-70df-45d5-b86b-8392a8bd8791 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:09:38.875472 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-aef510e8-70df-45d5-b86b-8392a8bd8791 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:09:38.876713 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-aef510e8-70df-45d5-b86b-8392a8bd8791 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:09:38.885562 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1458/2898] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:09:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:09:38.892421 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c72006a9-01f8-492c-8ed9-2428f17012b6 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:09:38.910083 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c72006a9-01f8-492c-8ed9-2428f17012b6 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:09:38.919598 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-123de653-291a-4153-97b2-fe82db77e936 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:09:38.920456 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123de653-291a-4153-97b2-fe82db77e936 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:09:38.920887 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123de653-291a-4153-97b2-fe82db77e936 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:09:38.921322 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123de653-291a-4153-97b2-fe82db77e936 admin admin] PATH_INFO: `/v3/domains` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:09:38.922316 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-123de653-291a-4153-97b2-fe82db77e936 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81386) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 01:09:38.922823 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-123de653-291a-4153-97b2-fe82db77e936 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:09:38.925724 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-123de653-291a-4153-97b2-fe82db77e936 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:09:38.950450 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1441/2899] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 01:09:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 01:09:38.955001 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aef510e8-70df-45d5-b86b-8392a8bd8791 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:09:38.965661 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aef510e8-70df-45d5-b86b-8392a8bd8791 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:09:38.971020 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10d2b656-5ceb-402a-a370-c6b0ab22d997 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:09:38.971711 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d2b656-5ceb-402a-a370-c6b0ab22d997 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:09:38.972033 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d2b656-5ceb-402a-a370-c6b0ab22d997 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:09:38.972348 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d2b656-5ceb-402a-a370-c6b0ab22d997 admin admin] PATH_INFO: `/v3/projects` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:09:38.973058 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10d2b656-5ceb-402a-a370-c6b0ab22d997 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:09:38.975531 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10d2b656-5ceb-402a-a370-c6b0ab22d997 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:09:39.007463 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1459/2900] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 01:09:38 2022] POST /identity/v3/projects => generated 375 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 01:09:39.012469 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-123de653-291a-4153-97b2-fe82db77e936 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:09:39.024156 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-123de653-291a-4153-97b2-fe82db77e936 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:09:39.086471 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:09:39.103545 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9377332-a1e2-4508-a502-0f7444856ea3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:09:39.104694 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9377332-a1e2-4508-a502-0f7444856ea3 admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:09:39.104997 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9377332-a1e2-4508-a502-0f7444856ea3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:09:39.105288 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9377332-a1e2-4508-a502-0f7444856ea3 admin admin] PATH_INFO: `/v3/users` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:09:39.106112 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9377332-a1e2-4508-a502-0f7444856ea3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:09:39.108234 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9377332-a1e2-4508-a502-0f7444856ea3 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:09:39.140685 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1442/2901] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 01:09:39 2022] POST /identity/v3/users => generated 288 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 01:09:39.145387 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10d2b656-5ceb-402a-a370-c6b0ab22d997 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:09:39.157242 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10d2b656-5ceb-402a-a370-c6b0ab22d997 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:09:39.162419 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ff00da6-8bbe-4383-a8d7-c7f7a4c9fa70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:09:39.163046 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff00da6-8bbe-4383-a8d7-c7f7a4c9fa70 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:09:39.163321 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff00da6-8bbe-4383-a8d7-c7f7a4c9fa70 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:09:39.163592 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff00da6-8bbe-4383-a8d7-c7f7a4c9fa70 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:09:39.164247 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ff00da6-8bbe-4383-a8d7-c7f7a4c9fa70 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:09:39.165916 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ff00da6-8bbe-4383-a8d7-c7f7a4c9fa70 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:09:39.178160 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1460/2902] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:09:39 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 01:09:39.185293 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9377332-a1e2-4508-a502-0f7444856ea3 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:09:39.206304 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9377332-a1e2-4508-a502-0f7444856ea3 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:09:39.215712 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6eb73ef-5bd9-4022-8ac5-e4c5b50548f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:09:39.216741 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6eb73ef-5bd9-4022-8ac5-e4c5b50548f8 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:09:39.217226 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6eb73ef-5bd9-4022-8ac5-e4c5b50548f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:09:39.217740 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6eb73ef-5bd9-4022-8ac5-e4c5b50548f8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:09:39.218804 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6eb73ef-5bd9-4022-8ac5-e4c5b50548f8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:09:39.221815 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6eb73ef-5bd9-4022-8ac5-e4c5b50548f8 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:09:39.235958 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1443/2903] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:09:39 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 01:09:39.242560 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ff00da6-8bbe-4383-a8d7-c7f7a4c9fa70 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:09:39.254491 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ff00da6-8bbe-4383-a8d7-c7f7a4c9fa70 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:09:39.260063 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0352a78-2e7c-415d-807c-ad4f7d615c24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:09:39.260746 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0352a78-2e7c-415d-807c-ad4f7d615c24 admin admin] REQUEST_METHOD: `PUT` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:09:39.261139 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0352a78-2e7c-415d-807c-ad4f7d615c24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:09:39.261464 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0352a78-2e7c-415d-807c-ad4f7d615c24 admin admin] PATH_INFO: `/v3/projects/46f8efea67784ad48d92d7ab27e04e5e/users/9e0e13e4786c4de395d51409dcb48cb5/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:09:39.279597 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0352a78-2e7c-415d-807c-ad4f7d615c24 admin admin] RBAC: Authorizing `identity:create_grant(project_id=46f8efea67784ad48d92d7ab27e04e5e, user_id=9e0e13e4786c4de395d51409dcb48cb5, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:09:39.282189 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0352a78-2e7c-415d-807c-ad4f7d615c24 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:09:39.299814 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1461/2904] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 01:09:39 2022] PUT /identity/v3/projects/46f8efea67784ad48d92d7ab27e04e5e/users/9e0e13e4786c4de395d51409dcb48cb5/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:09:39.312370 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea021c9-5b0c-45c4-96c3-eab9e33ff848 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:09:39.312742 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea021c9-5b0c-45c4-96c3-eab9e33ff848 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:09:39.313041 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea021c9-5b0c-45c4-96c3-eab9e33ff848 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:09:39.347094 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-0ea021c9-5b0c-45c4-96c3-eab9e33ff848 None None] MFA Rules not processed for user `9e0e13e4786c4de395d51409dcb48cb5`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:09:39.348443 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-0ea021c9-5b0c-45c4-96c3-eab9e33ff848 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:09:39.434879 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1444/2905] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:09:39 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 01:09:39.447090 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0352a78-2e7c-415d-807c-ad4f7d615c24 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:09:39.450684 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-b0352a78-2e7c-415d-807c-ad4f7d615c24 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:09:39.469838 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:09:39.539664 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0352a78-2e7c-415d-807c-ad4f7d615c24 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:09:39.544945 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60781378-0e25-45b4-afec-958a5e32151c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:09:39.545821 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60781378-0e25-45b4-afec-958a5e32151c service cinder] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:09:39.546219 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60781378-0e25-45b4-afec-958a5e32151c service cinder] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:09:39.546611 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60781378-0e25-45b4-afec-958a5e32151c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:09:39.556492 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60781378-0e25-45b4-afec-958a5e32151c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:09:39.558582 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60781378-0e25-45b4-afec-958a5e32151c service cinder] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:09:39.571444 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1462/2906] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:09:39 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:09:40.991525 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bac6710-fb2e-4c78-ab99-7bd9754a5586 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:09:40.992156 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bac6710-fb2e-4c78-ab99-7bd9754a5586 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:09:40.992156 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bac6710-fb2e-4c78-ab99-7bd9754a5586 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:09:41.023100 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-0bac6710-fb2e-4c78-ab99-7bd9754a5586 None None] MFA Rules not processed for user `9e0e13e4786c4de395d51409dcb48cb5`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:09:41.024469 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-0bac6710-fb2e-4c78-ab99-7bd9754a5586 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:09:41.032527 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1445/2907] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:09:40 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:09:49.564274 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=23 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:09:51.016572 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=48,keystone:INSERT=3 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:09:57.785110 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94886a8d-3059-49b2-bf25-cffd1bca1c2f None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:09:57.785812 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94886a8d-3059-49b2-bf25-cffd1bca1c2f None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:09:57.785812 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94886a8d-3059-49b2-bf25-cffd1bca1c2f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:09:57.807549 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-94886a8d-3059-49b2-bf25-cffd1bca1c2f None None] MFA Rules not processed for user `3f20ffd8df994c1db1b1e7248db5b5a3`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:09:57.808850 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-94886a8d-3059-49b2-bf25-cffd1bca1c2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:09:57.903254 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1463/2908] 10.222.0.52 () {58 vars in 1009 bytes} [Sun Sep 25 01:09:57 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 01:09:57.916519 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bac6710-fb2e-4c78-ab99-7bd9754a5586 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:09:57.918160 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-0bac6710-fb2e-4c78-ab99-7bd9754a5586 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:09:57.999602 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bac6710-fb2e-4c78-ab99-7bd9754a5586 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:09:58.005205 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97316c5-cf7a-4348-ab1b-db1bfe43d66b service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da606b3906dd4171bd76fd281e03e258', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:09:58.005925 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97316c5-cf7a-4348-ab1b-db1bfe43d66b service swift] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:09:58.006314 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97316c5-cf7a-4348-ab1b-db1bfe43d66b service swift] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:09:58.006588 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97316c5-cf7a-4348-ab1b-db1bfe43d66b service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:09:58.018126 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f97316c5-cf7a-4348-ab1b-db1bfe43d66b service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:09:58.019945 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f97316c5-cf7a-4348-ab1b-db1bfe43d66b service swift] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:09:58.030497 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1446/2909] 10.222.0.52 () {60 vars in 1384 bytes} [Sun Sep 25 01:09:57 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 25 01:09:58.664091 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18debdcf-ba04-4538-acb0-c20c2bf78281 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:09:58.664500 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18debdcf-ba04-4538-acb0-c20c2bf78281 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:09:58.664810 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18debdcf-ba04-4538-acb0-c20c2bf78281 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:09:58.687901 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-18debdcf-ba04-4538-acb0-c20c2bf78281 None None] MFA Rules not processed for user `3f20ffd8df994c1db1b1e7248db5b5a3`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:09:58.689239 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-18debdcf-ba04-4538-acb0-c20c2bf78281 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:09:58.696976 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1464/2910] 10.222.0.52 () {58 vars in 967 bytes} [Sun Sep 25 01:09:58 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 01:09:58.710516 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97316c5-cf7a-4348-ab1b-db1bfe43d66b service swift] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:09:58.723183 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97316c5-cf7a-4348-ab1b-db1bfe43d66b service swift] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:09:58.731450 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf556db8-88f0-49cf-acb8-b910595bc642 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da606b3906dd4171bd76fd281e03e258', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:09:58.732084 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf556db8-88f0-49cf-acb8-b910595bc642 service swift] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:09:58.732376 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf556db8-88f0-49cf-acb8-b910595bc642 service swift] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:09:58.732622 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf556db8-88f0-49cf-acb8-b910595bc642 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:09:58.743725 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf556db8-88f0-49cf-acb8-b910595bc642 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:09:58.747431 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf556db8-88f0-49cf-acb8-b910595bc642 service swift] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:09:58.764298 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1447/2911] 10.222.0.52 () {60 vars in 1384 bytes} [Sun Sep 25 01:09:58 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 25 01:09:59.059572 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ee2feb-b5cd-40d4-8885-9e623f032c7e None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:09:59.060167 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ee2feb-b5cd-40d4-8885-9e623f032c7e None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:09:59.060393 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ee2feb-b5cd-40d4-8885-9e623f032c7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:09:59.082630 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-78ee2feb-b5cd-40d4-8885-9e623f032c7e None None] MFA Rules not processed for user `3f20ffd8df994c1db1b1e7248db5b5a3`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:09:59.085626 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-78ee2feb-b5cd-40d4-8885-9e623f032c7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:09:59.094495 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1465/2912] 10.222.0.52 () {58 vars in 1009 bytes} [Sun Sep 25 01:09:59 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 01:09:59.109090 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf556db8-88f0-49cf-acb8-b910595bc642 service swift] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:09:59.122250 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf556db8-88f0-49cf-acb8-b910595bc642 service swift] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:09:59.128224 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c183bd-999b-4c90-a353-fe8c0b24d8a7 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da606b3906dd4171bd76fd281e03e258', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:09:59.128841 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c183bd-999b-4c90-a353-fe8c0b24d8a7 service swift] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:09:59.129114 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c183bd-999b-4c90-a353-fe8c0b24d8a7 service swift] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:09:59.129420 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c183bd-999b-4c90-a353-fe8c0b24d8a7 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:09:59.139304 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7c183bd-999b-4c90-a353-fe8c0b24d8a7 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:09:59.141096 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7c183bd-999b-4c90-a353-fe8c0b24d8a7 service swift] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:09:59.151461 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1448/2913] 10.222.0.52 () {60 vars in 1384 bytes} [Sun Sep 25 01:09:59 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 25 01:09:59.530721 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d04d302b-54a6-4629-9c13-c190fff7cd30 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:09:59.531155 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d04d302b-54a6-4629-9c13-c190fff7cd30 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:09:59.531551 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d04d302b-54a6-4629-9c13-c190fff7cd30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:09:59.554988 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-d04d302b-54a6-4629-9c13-c190fff7cd30 None None] MFA Rules not processed for user `3f20ffd8df994c1db1b1e7248db5b5a3`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:09:59.556371 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-d04d302b-54a6-4629-9c13-c190fff7cd30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:09:59.564165 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1466/2914] 10.222.0.52 () {58 vars in 967 bytes} [Sun Sep 25 01:09:59 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:09:59.581582 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c183bd-999b-4c90-a353-fe8c0b24d8a7 service swift] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:09:59.593966 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c183bd-999b-4c90-a353-fe8c0b24d8a7 service swift] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:09:59.599367 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a25b5bf-9cea-49ee-ae38-3106a5a2eeeb service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da606b3906dd4171bd76fd281e03e258', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:09:59.599985 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a25b5bf-9cea-49ee-ae38-3106a5a2eeeb service swift] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:09:59.600265 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a25b5bf-9cea-49ee-ae38-3106a5a2eeeb service swift] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:09:59.600537 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a25b5bf-9cea-49ee-ae38-3106a5a2eeeb service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:09:59.611195 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a25b5bf-9cea-49ee-ae38-3106a5a2eeeb service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:09:59.612833 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a25b5bf-9cea-49ee-ae38-3106a5a2eeeb service swift] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:09:59.623706 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1449/2915] 10.222.0.52 () {60 vars in 1384 bytes} [Sun Sep 25 01:09:59 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 25 01:10:01.909188 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d04d302b-54a6-4629-9c13-c190fff7cd30 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:10:01.986003 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d04d302b-54a6-4629-9c13-c190fff7cd30 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:10:01.990945 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd43c69-f493-4ece-95ec-8552a73be68e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:10:01.991750 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd43c69-f493-4ece-95ec-8552a73be68e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:10:01.992057 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd43c69-f493-4ece-95ec-8552a73be68e admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:10:01.992327 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd43c69-f493-4ece-95ec-8552a73be68e admin admin] PATH_INFO: `/v3/users/9e0e13e4786c4de395d51409dcb48cb5` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:10:01.993527 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccd43c69-f493-4ece-95ec-8552a73be68e admin admin] RBAC: Authorizing `identity:delete_user(user_id=9e0e13e4786c4de395d51409dcb48cb5)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:10:01.995622 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccd43c69-f493-4ece-95ec-8552a73be68e admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:10:02.048007 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-ccd43c69-f493-4ece-95ec-8552a73be68e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9e0e13e4786c4de395d51409dcb48cb5'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:10:02.048663 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-ccd43c69-f493-4ece-95ec-8552a73be68e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9e0e13e4786c4de395d51409dcb48cb5'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:10:02.056603 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-ccd43c69-f493-4ece-95ec-8552a73be68e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9e0e13e4786c4de395d51409dcb48cb5'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:10:02.057628 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-ccd43c69-f493-4ece-95ec-8552a73be68e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9e0e13e4786c4de395d51409dcb48cb5'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:10:02.065247 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-ccd43c69-f493-4ece-95ec-8552a73be68e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9e0e13e4786c4de395d51409dcb48cb5'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:10:02.085375 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1467/2916] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 01:10:01 2022] DELETE /identity/v3/users/9e0e13e4786c4de395d51409dcb48cb5 => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:10:02.090558 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9edcf439-9aef-40d3-ad07-a6bd60fdcd74 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:10:02.091051 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9edcf439-9aef-40d3-ad07-a6bd60fdcd74 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:10:02.091165 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9edcf439-9aef-40d3-ad07-a6bd60fdcd74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:10:02.110484 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-9edcf439-9aef-40d3-ad07-a6bd60fdcd74 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:10:02.111702 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-9edcf439-9aef-40d3-ad07-a6bd60fdcd74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:10:02.172619 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1450/2917] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:10:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:10:02.182583 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd43c69-f493-4ece-95ec-8552a73be68e admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:10:02.184005 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-ccd43c69-f493-4ece-95ec-8552a73be68e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:10:02.191130 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:10:02.247701 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd43c69-f493-4ece-95ec-8552a73be68e admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:10:02.251444 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb52970e-45af-4132-8c85-2bd05c0cb342 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:10:02.252148 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb52970e-45af-4132-8c85-2bd05c0cb342 service neutron] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:10:02.252322 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb52970e-45af-4132-8c85-2bd05c0cb342 service neutron] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:10:02.252672 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb52970e-45af-4132-8c85-2bd05c0cb342 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:10:02.263317 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb52970e-45af-4132-8c85-2bd05c0cb342 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:10:02.264755 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb52970e-45af-4132-8c85-2bd05c0cb342 service neutron] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:10:02.279849 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1468/2918] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:10:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:10:02.472307 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9edcf439-9aef-40d3-ad07-a6bd60fdcd74 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:10:02.474749 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-9edcf439-9aef-40d3-ad07-a6bd60fdcd74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:10:02.492862 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9edcf439-9aef-40d3-ad07-a6bd60fdcd74 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:10:02.501382 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606e436b-fa50-415e-8500-ce832bdb1272 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:10:02.502370 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606e436b-fa50-415e-8500-ce832bdb1272 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:10:02.502922 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606e436b-fa50-415e-8500-ce832bdb1272 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:10:02.503275 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606e436b-fa50-415e-8500-ce832bdb1272 admin admin] PATH_INFO: `/v3/projects/46f8efea67784ad48d92d7ab27e04e5e` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:10:02.515218 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-606e436b-fa50-415e-8500-ce832bdb1272 admin admin] RBAC: Authorizing `identity:delete_project(project_id=46f8efea67784ad48d92d7ab27e04e5e)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:10:02.517212 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-606e436b-fa50-415e-8500-ce832bdb1272 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:10:02.568721 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:10:02.572895 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-606e436b-fa50-415e-8500-ce832bdb1272 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '46f8efea67784ad48d92d7ab27e04e5e', 'request_id': 'req-606e436b-fa50-415e-8500-ce832bdb1272'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:10:02.582087 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-606e436b-fa50-415e-8500-ce832bdb1272 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '46f8efea67784ad48d92d7ab27e04e5e', 'request_id': 'req-606e436b-fa50-415e-8500-ce832bdb1272'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:10:02.589495 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-606e436b-fa50-415e-8500-ce832bdb1272 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '46f8efea67784ad48d92d7ab27e04e5e', 'request_id': 'req-606e436b-fa50-415e-8500-ce832bdb1272'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:10:02.610117 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-606e436b-fa50-415e-8500-ce832bdb1272 admin admin] The token cache is being invalidate because project 46f8efea67784ad48d92d7ab27e04e5e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81386) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 01:10:02.610554 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-606e436b-fa50-415e-8500-ce832bdb1272 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:10:02.611221 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-606e436b-fa50-415e-8500-ce832bdb1272 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:10:02.612722 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1451/2919] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 01:10:02 2022] DELETE /identity/v3/projects/46f8efea67784ad48d92d7ab27e04e5e => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:10:02.617890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5add55bb-edd6-4c4d-b0d1-839fda68a624 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:10:02.618172 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5add55bb-edd6-4c4d-b0d1-839fda68a624 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:10:02.618440 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5add55bb-edd6-4c4d-b0d1-839fda68a624 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:10:02.640094 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-5add55bb-edd6-4c4d-b0d1-839fda68a624 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:10:02.641314 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-5add55bb-edd6-4c4d-b0d1-839fda68a624 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:10:02.703998 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1469/2920] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:10:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:10:02.713229 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-218447ed-aa3f-4a54-a7c2-fa68b05662c4 None None] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:10:02.713678 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-218447ed-aa3f-4a54-a7c2-fa68b05662c4 None None] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:10:02.714074 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-218447ed-aa3f-4a54-a7c2-fa68b05662c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:10:02.739220 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.auth.core [None req-218447ed-aa3f-4a54-a7c2-fa68b05662c4 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81386) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:10:02.740436 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-218447ed-aa3f-4a54-a7c2-fa68b05662c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:10:02.749321 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1452/2921] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:10:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 01:10:02.753679 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5add55bb-edd6-4c4d-b0d1-839fda68a624 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:10:02.764555 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5add55bb-edd6-4c4d-b0d1-839fda68a624 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:10:02.769505 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb1fead4-f160-4022-b7ff-85d142746dfb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:10:02.770367 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb1fead4-f160-4022-b7ff-85d142746dfb admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:10:02.770367 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb1fead4-f160-4022-b7ff-85d142746dfb admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:10:02.770647 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb1fead4-f160-4022-b7ff-85d142746dfb admin admin] PATH_INFO: `/v3/domains` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:10:02.771301 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb1fead4-f160-4022-b7ff-85d142746dfb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81387) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 01:10:02.771601 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb1fead4-f160-4022-b7ff-85d142746dfb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:10:02.773214 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb1fead4-f160-4022-b7ff-85d142746dfb admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:10:02.786503 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1470/2922] 10.222.0.52 () {58 vars in 1153 bytes} [Sun Sep 25 01:10:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 01:10:02.791630 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-218447ed-aa3f-4a54-a7c2-fa68b05662c4 None None] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:10:02.801172 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-218447ed-aa3f-4a54-a7c2-fa68b05662c4 None None] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:10:02.806037 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-043e3e14-9d12-4f8f-805e-77962293db6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:10:02.807499 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043e3e14-9d12-4f8f-805e-77962293db6c admin admin] REQUEST_METHOD: `POST` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:10:02.807776 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043e3e14-9d12-4f8f-805e-77962293db6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:10:02.808031 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043e3e14-9d12-4f8f-805e-77962293db6c admin admin] PATH_INFO: `/v3/projects` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:10:02.808668 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-043e3e14-9d12-4f8f-805e-77962293db6c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:10:02.810608 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-043e3e14-9d12-4f8f-805e-77962293db6c admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:10:02.841136 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1453/2923] 10.222.0.52 () {60 vars in 1153 bytes} [Sun Sep 25 01:10:02 2022] POST /identity/v3/projects => generated 383 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 01:10:02.846860 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb1fead4-f160-4022-b7ff-85d142746dfb admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:10:02.858160 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb1fead4-f160-4022-b7ff-85d142746dfb admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:10:02.926940 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096ff613-6578-48de-bd01-9d678e810541 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:10:02.927649 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096ff613-6578-48de-bd01-9d678e810541 admin admin] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:10:02.927811 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096ff613-6578-48de-bd01-9d678e810541 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:10:02.928078 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096ff613-6578-48de-bd01-9d678e810541 admin admin] PATH_INFO: `/v3/users` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:10:02.928715 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096ff613-6578-48de-bd01-9d678e810541 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:10:02.930649 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096ff613-6578-48de-bd01-9d678e810541 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:10:02.957959 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1471/2924] 10.222.0.52 () {60 vars in 1144 bytes} [Sun Sep 25 01:10:02 2022] POST /identity/v3/users => generated 292 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 01:10:02.962070 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-043e3e14-9d12-4f8f-805e-77962293db6c admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:10:02.973879 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-043e3e14-9d12-4f8f-805e-77962293db6c admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:10:02.979741 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f690ef-f2f2-4ecf-9b17-c4dbe2d38972 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:10:02.980387 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f690ef-f2f2-4ecf-9b17-c4dbe2d38972 admin admin] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:10:02.980664 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f690ef-f2f2-4ecf-9b17-c4dbe2d38972 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:10:02.980932 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f690ef-f2f2-4ecf-9b17-c4dbe2d38972 admin admin] PATH_INFO: `/v3/roles` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:10:02.981949 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42f690ef-f2f2-4ecf-9b17-c4dbe2d38972 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:10:02.983554 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42f690ef-f2f2-4ecf-9b17-c4dbe2d38972 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:10:02.996401 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1454/2925] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:10:02 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 01:10:03.000737 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096ff613-6578-48de-bd01-9d678e810541 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:10:03.019601 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096ff613-6578-48de-bd01-9d678e810541 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:10:03.026499 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27040ab5-ce4c-402a-ae34-5dfe9b027860 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:10:03.027164 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27040ab5-ce4c-402a-ae34-5dfe9b027860 admin admin] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:10:03.027444 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27040ab5-ce4c-402a-ae34-5dfe9b027860 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:10:03.027805 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27040ab5-ce4c-402a-ae34-5dfe9b027860 admin admin] PATH_INFO: `/v3/roles` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:10:03.028507 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27040ab5-ce4c-402a-ae34-5dfe9b027860 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:10:03.030240 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27040ab5-ce4c-402a-ae34-5dfe9b027860 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:10:03.043231 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1472/2926] 10.222.0.52 () {58 vars in 1122 bytes} [Sun Sep 25 01:10:02 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 01:10:03.047747 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f690ef-f2f2-4ecf-9b17-c4dbe2d38972 admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:10:03.059314 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f690ef-f2f2-4ecf-9b17-c4dbe2d38972 admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:10:03.064203 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d02dca94-b255-4064-a8b8-67abe887ae7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:10:03.064886 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02dca94-b255-4064-a8b8-67abe887ae7c admin admin] REQUEST_METHOD: `PUT` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:10:03.065147 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02dca94-b255-4064-a8b8-67abe887ae7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:10:03.065506 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02dca94-b255-4064-a8b8-67abe887ae7c admin admin] PATH_INFO: `/v3/projects/1f0ea61e4de94ce083c572843a3f09fa/users/d978fab9ac0a4094a7e1008337ed2af9/roles/40f54f5ac0994c51960ac413da8d2cc9` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:10:03.084032 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d02dca94-b255-4064-a8b8-67abe887ae7c admin admin] RBAC: Authorizing `identity:create_grant(project_id=1f0ea61e4de94ce083c572843a3f09fa, user_id=d978fab9ac0a4094a7e1008337ed2af9, role_id=40f54f5ac0994c51960ac413da8d2cc9)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:10:03.086791 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d02dca94-b255-4064-a8b8-67abe887ae7c admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:10:03.102143 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1455/2927] 10.222.0.52 () {60 vars in 1483 bytes} [Sun Sep 25 01:10:03 2022] PUT /identity/v3/projects/1f0ea61e4de94ce083c572843a3f09fa/users/d978fab9ac0a4094a7e1008337ed2af9/roles/40f54f5ac0994c51960ac413da8d2cc9 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:10:03.108469 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a5ba75-2d8f-43b4-9695-d420730266d2 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:10:03.108744 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a5ba75-2d8f-43b4-9695-d420730266d2 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:10:03.109015 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a5ba75-2d8f-43b4-9695-d420730266d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:10:03.136181 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-76a5ba75-2d8f-43b4-9695-d420730266d2 None None] MFA Rules not processed for user `d6d73e23626043529ac5b9e3889933d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:10:03.137691 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-76a5ba75-2d8f-43b4-9695-d420730266d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:10:03.147268 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:10:03.208409 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1473/2928] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:10:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:10:03.219167 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d02dca94-b255-4064-a8b8-67abe887ae7c admin admin] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:10:03.220902 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-d02dca94-b255-4064-a8b8-67abe887ae7c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:10:03.292035 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d02dca94-b255-4064-a8b8-67abe887ae7c admin admin] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:10:03.295958 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b5bc07a-c8e0-48dd-afad-ad35188c910d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:10:03.296596 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5bc07a-c8e0-48dd-afad-ad35188c910d service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:10:03.296869 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5bc07a-c8e0-48dd-afad-ad35188c910d service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:10:03.297214 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5bc07a-c8e0-48dd-afad-ad35188c910d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:10:03.307884 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b5bc07a-c8e0-48dd-afad-ad35188c910d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:10:03.309354 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b5bc07a-c8e0-48dd-afad-ad35188c910d service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:10:03.320458 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1456/2929] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:10:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:10:03.999695 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a3bc02-fe8a-49a9-a66d-98c5d813dd58 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:10:04.000215 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a3bc02-fe8a-49a9-a66d-98c5d813dd58 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:10:04.000362 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a3bc02-fe8a-49a9-a66d-98c5d813dd58 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:10:04.032059 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-23a3bc02-fe8a-49a9-a66d-98c5d813dd58 None None] MFA Rules not processed for user `d978fab9ac0a4094a7e1008337ed2af9`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:10:04.033248 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-23a3bc02-fe8a-49a9-a66d-98c5d813dd58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:10:04.118311 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1474/2930] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:10:03 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:10:04.132076 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b5bc07a-c8e0-48dd-afad-ad35188c910d service neutron] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:10:04.135298 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-9b5bc07a-c8e0-48dd-afad-ad35188c910d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:10:04.203678 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b5bc07a-c8e0-48dd-afad-ad35188c910d service neutron] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:10:04.207788 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc265e1-794e-4403-8632-f612edc52fed service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:10:04.208433 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc265e1-794e-4403-8632-f612edc52fed service cinder] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:10:04.208693 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc265e1-794e-4403-8632-f612edc52fed service cinder] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:10:04.208987 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc265e1-794e-4403-8632-f612edc52fed service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:10:04.219833 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbc265e1-794e-4403-8632-f612edc52fed service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:10:04.221376 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbc265e1-794e-4403-8632-f612edc52fed service cinder] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:10:04.233518 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1457/2931] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:10:04 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:10:05.671963 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd693709-1036-4fef-9ecc-7ec4d414b68e None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:10:05.672571 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd693709-1036-4fef-9ecc-7ec4d414b68e None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:10:05.672737 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd693709-1036-4fef-9ecc-7ec4d414b68e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:10:05.703811 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-cd693709-1036-4fef-9ecc-7ec4d414b68e None None] MFA Rules not processed for user `d978fab9ac0a4094a7e1008337ed2af9`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:10:05.705899 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-cd693709-1036-4fef-9ecc-7ec4d414b68e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:10:05.717890 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1475/2932] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:10:05 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:10:06.007616 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc265e1-794e-4403-8632-f612edc52fed service cinder] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:10:06.007616 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-cbc265e1-794e-4403-8632-f612edc52fed service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:10:06.023590 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc265e1-794e-4403-8632-f612edc52fed service cinder] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:10:06.027956 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f053f630-5d61-4b48-bf16-60e4c9be0640 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:10:06.028566 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f053f630-5d61-4b48-bf16-60e4c9be0640 service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:10:06.029039 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f053f630-5d61-4b48-bf16-60e4c9be0640 service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:10:06.029523 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f053f630-5d61-4b48-bf16-60e4c9be0640 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:10:06.031659 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-f053f630-5d61-4b48-bf16-60e4c9be0640 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:10:06.045049 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f053f630-5d61-4b48-bf16-60e4c9be0640 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:10:06.046559 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f053f630-5d61-4b48-bf16-60e4c9be0640 service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:10:06.058656 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1458/2933] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:10:06 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:10:15.694620 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=56 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:10:16.052138 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:DELETE=4,keystone:SELECT=92,keystone:INSERT=3 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:10:58.173421 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd693709-1036-4fef-9ecc-7ec4d414b68e None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:10:58.175491 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-cd693709-1036-4fef-9ecc-7ec4d414b68e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:10:58.189509 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd693709-1036-4fef-9ecc-7ec4d414b68e None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:10:58.193477 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939d0815-fe58-4996-b455-0de7902850b6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '944f10f38ee2428a85599f6a714ac806', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:10:58.194123 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939d0815-fe58-4996-b455-0de7902850b6 service cinder] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:10:58.194396 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939d0815-fe58-4996-b455-0de7902850b6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:10:58.194672 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939d0815-fe58-4996-b455-0de7902850b6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:10:58.197107 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-939d0815-fe58-4996-b455-0de7902850b6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:10:58.283397 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-939d0815-fe58-4996-b455-0de7902850b6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:10:58.284867 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-939d0815-fe58-4996-b455-0de7902850b6 service cinder] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:10:58.298875 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1476/2934] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:10:58 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:11:01.018399 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f053f630-5d61-4b48-bf16-60e4c9be0640 service neutron] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:11:01.031547 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f053f630-5d61-4b48-bf16-60e4c9be0640 service neutron] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:11:01.037801 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9b93b57-c84e-4a99-8bd1-681952906af5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '20c274e96dec478b98ef51f80ea893b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:11:01.038828 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b93b57-c84e-4a99-8bd1-681952906af5 service neutron] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:11:01.039880 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b93b57-c84e-4a99-8bd1-681952906af5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:11:01.040361 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b93b57-c84e-4a99-8bd1-681952906af5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:11:01.043500 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-f9b93b57-c84e-4a99-8bd1-681952906af5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:11:01.065651 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9b93b57-c84e-4a99-8bd1-681952906af5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:11:01.067149 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9b93b57-c84e-4a99-8bd1-681952906af5 service neutron] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:11:01.080001 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1459/2935] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:11:01 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 62 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:11:08.292495 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=34 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:11:11.074259 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=6 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:11:11.385222 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39332254-ec67-4720-8a75-32102665b530 None None] REQUEST_METHOD: `POST` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:11:11.385853 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39332254-ec67-4720-8a75-32102665b530 None None] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:11:11.385853 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39332254-ec67-4720-8a75-32102665b530 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:11:11.407237 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.auth.core [None req-39332254-ec67-4720-8a75-32102665b530 None None] MFA Rules not processed for user `d978fab9ac0a4094a7e1008337ed2af9`. Rule list: `[]` (Enabled: `True`). {{(pid=81387) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 01:11:11.408553 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-39332254-ec67-4720-8a75-32102665b530 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:11:11.417115 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1477/2936] 10.222.0.52 () {58 vars in 958 bytes} [Sun Sep 25 01:11:11 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 01:11:21.400967 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=3 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:11:48.520491 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9b93b57-c84e-4a99-8bd1-681952906af5 service neutron] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:11:48.525566 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-f9b93b57-c84e-4a99-8bd1-681952906af5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:11:48.544612 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9b93b57-c84e-4a99-8bd1-681952906af5 service neutron] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:11:48.551195 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa38ec5-65bd-47ab-af37-855fb2ea313e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f142988a4dd14a5f96eacea45c046f85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:11:48.551810 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa38ec5-65bd-47ab-af37-855fb2ea313e service nova] REQUEST_METHOD: `GET` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:11:48.552417 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa38ec5-65bd-47ab-af37-855fb2ea313e service nova] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:11:48.552865 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa38ec5-65bd-47ab-af37-855fb2ea313e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:11:48.555697 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.fernet_utils [None req-7fa38ec5-65bd-47ab-af37-855fb2ea313e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81386) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:11:48.576224 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa38ec5-65bd-47ab-af37-855fb2ea313e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:11:48.578042 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa38ec5-65bd-47ab-af37-855fb2ea313e service nova] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:11:48.591456 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1460/2937] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:11:48 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 73 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:11:58.585126 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=6 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:12:13.397287 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39332254-ec67-4720-8a75-32102665b530 None None] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:12:13.410450 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39332254-ec67-4720-8a75-32102665b530 None None] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:12:13.416549 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4499af6-85b5-4de2-877c-9c2bd32ad944 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:12:13.417553 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4499af6-85b5-4de2-877c-9c2bd32ad944 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:12:13.417825 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4499af6-85b5-4de2-877c-9c2bd32ad944 admin admin] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:12:13.418097 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4499af6-85b5-4de2-877c-9c2bd32ad944 admin admin] PATH_INFO: `/v3/users/d978fab9ac0a4094a7e1008337ed2af9` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:12:13.419296 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4499af6-85b5-4de2-877c-9c2bd32ad944 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d978fab9ac0a4094a7e1008337ed2af9)` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:12:13.421271 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4499af6-85b5-4de2-877c-9c2bd32ad944 admin admin] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:12:13.483661 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-b4499af6-85b5-4de2-877c-9c2bd32ad944 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd978fab9ac0a4094a7e1008337ed2af9'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:12:13.484427 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-b4499af6-85b5-4de2-877c-9c2bd32ad944 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd978fab9ac0a4094a7e1008337ed2af9'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:12:13.493864 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-b4499af6-85b5-4de2-877c-9c2bd32ad944 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd978fab9ac0a4094a7e1008337ed2af9'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:12:13.494512 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-b4499af6-85b5-4de2-877c-9c2bd32ad944 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd978fab9ac0a4094a7e1008337ed2af9'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:12:13.504628 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.notifications [None req-b4499af6-85b5-4de2-877c-9c2bd32ad944 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd978fab9ac0a4094a7e1008337ed2af9'} {{(pid=81387) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:12:13.526364 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1478/2938] 10.222.0.52 () {58 vars in 1224 bytes} [Sun Sep 25 01:12:13 2022] DELETE /identity/v3/users/d978fab9ac0a4094a7e1008337ed2af9 => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 01:12:13.739675 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa38ec5-65bd-47ab-af37-855fb2ea313e service nova] Authenticating user token {{(pid=81386) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:12:13.754055 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa38ec5-65bd-47ab-af37-855fb2ea313e service nova] Validating token access rules against request {{(pid=81386) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:12:13.827920 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69a438a-1ca0-457b-be4c-72d438a02ae7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6d73e23626043529ac5b9e3889933d4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c8e32a9668254c2e96354fd797883364', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81386) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:12:13.828584 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69a438a-1ca0-457b-be4c-72d438a02ae7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:12:13.828857 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69a438a-1ca0-457b-be4c-72d438a02ae7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:12:13.829191 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69a438a-1ca0-457b-be4c-72d438a02ae7 admin admin] PATH_INFO: `/v3/projects/1f0ea61e4de94ce083c572843a3f09fa` {{(pid=81386) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:12:13.830489 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d69a438a-1ca0-457b-be4c-72d438a02ae7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1f0ea61e4de94ce083c572843a3f09fa)` {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:12:13.832444 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d69a438a-1ca0-457b-be4c-72d438a02ae7 admin admin] RBAC: Authorization granted {{(pid=81386) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:12:13.890903 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-d69a438a-1ca0-457b-be4c-72d438a02ae7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1f0ea61e4de94ce083c572843a3f09fa', 'request_id': 'req-d69a438a-1ca0-457b-be4c-72d438a02ae7'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:12:13.900698 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-d69a438a-1ca0-457b-be4c-72d438a02ae7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1f0ea61e4de94ce083c572843a3f09fa', 'request_id': 'req-d69a438a-1ca0-457b-be4c-72d438a02ae7'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:12:13.909018 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-d69a438a-1ca0-457b-be4c-72d438a02ae7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1f0ea61e4de94ce083c572843a3f09fa', 'request_id': 'req-d69a438a-1ca0-457b-be4c-72d438a02ae7'} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:12:13.925015 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-d69a438a-1ca0-457b-be4c-72d438a02ae7 admin admin] The token cache is being invalidate because project 1f0ea61e4de94ce083c572843a3f09fa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81386) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 01:12:13.925414 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-d69a438a-1ca0-457b-be4c-72d438a02ae7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:12:13.926173 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG keystone.notifications [None req-d69a438a-1ca0-457b-be4c-72d438a02ae7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81386) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 01:12:13.927928 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: [pid: 81386|app: 0|req: 1461/2939] 10.222.0.52 () {58 vars in 1233 bytes} [Sun Sep 25 01:12:13 2022] DELETE /identity/v3/projects/1f0ea61e4de94ce083c572843a3f09fa => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 01:12:23.521865 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=29,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:12:23.920772 dv-cinder-master-3-iscsi devstack@keystone.service[81386]: DEBUG dbcounter [-] [81386] Writing DB stats keystone:SELECT=51,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81386) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 01:15:04.229442 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4499af6-85b5-4de2-877c-9c2bd32ad944 admin admin] Authenticating user token {{(pid=81387) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 01:15:04.231252 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-b4499af6-85b5-4de2-877c-9c2bd32ad944 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:15:04.421978 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4499af6-85b5-4de2-877c-9c2bd32ad944 admin admin] Validating token access rules against request {{(pid=81387) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 01:15:04.496108 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-997bcd9f-7616-49d0-8dc1-8823eccd0049 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '11a80b55dcd444df901c8bf80f2a2c26', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9db0fd07fc494203b2ada13f3cd046d3', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81387) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 01:15:04.496851 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997bcd9f-7616-49d0-8dc1-8823eccd0049 service placement] REQUEST_METHOD: `GET` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 01:15:04.497182 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997bcd9f-7616-49d0-8dc1-8823eccd0049 service placement] SCRIPT_NAME: `/identity` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 01:15:04.497565 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997bcd9f-7616-49d0-8dc1-8823eccd0049 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81387) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 01:15:04.499950 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.fernet_utils [None req-997bcd9f-7616-49d0-8dc1-8823eccd0049 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81387) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 01:15:04.515530 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-997bcd9f-7616-49d0-8dc1-8823eccd0049 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 01:15:04.518021 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-997bcd9f-7616-49d0-8dc1-8823eccd0049 service placement] RBAC: Authorization granted {{(pid=81387) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 01:15:04.529750 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: [pid: 81387|app: 0|req: 1479/2940] 10.222.0.52 () {60 vars in 1365 bytes} [Sun Sep 25 01:15:04 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 301 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 01:15:14.525049 dv-cinder-master-3-iscsi devstack@keystone.service[81387]: DEBUG dbcounter [-] [81387] Writing DB stats keystone:SELECT=89 {{(pid=81387) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}